Code

Stream ModelOutputs#1

Code

Closed

dmarx wants to merge 41 commits intomaincoreweave/transformers:mainfrom dmarx.output_streamercoreweave/transformers:dmarx.output_streamerCopy head branch name to clipboard

+560-121

Commits

Commits on Feb 29, 2024

• skeleton OutputStreamer

  dmarx

  committed
  f901ee0

• hacky but passes tests

  dmarx

  committed
  57a77d1

Commits on Mar 1, 2024

• skeleton OutputIteratorStreamer, notes

  dmarx

  committed
  b34c0b2

• fleshed out OutputIteratorStreamer w test case

  dmarx

  committed
  5eb0e69

• token_id streaming passes

  dmarx

  committed
  ff35e2b

• contrastive passes token_id test

  dmarx

  committed
  76ed5c2

• randomize test seed

  dmarx

  committed
  3885ea1

• greedy scores pass test

  dmarx

  committed
  2958099

Commits on Mar 2, 2024

• ensure we're building the output incrementally

  dmarx

  committed
  976eb23

Commits on Mar 4, 2024

• multinom sampling outputs match

  dmarx

  committed
  cabb5fe

Commits on Mar 5, 2024

• throw error if on_ready called on empty buffer

  dmarx

  committed
  9d075c9

Commits on Mar 6, 2024

• enforce list(values) in on_ready()

  dmarx

  committed
  ba48d71

• POC output_constructor

  dmarx

  committed
  108d3c2

• DRY tests following on_ready() fix

  dmarx

  committed
  02ec905

• fix function spelling

  dmarx

  committed
  ebe4798

• moved output_constructor to class method

  dmarx

  committed
  528d79a

• rename output_constructor -> _prepare_output

  dmarx

  committed
  97812bf

• integrating _prepare_output

  dmarx

  committed
  2c03ae6

• finished integrating _prepare_output

  dmarx

  committed
  64732bb

• placeholder args for attention/hidden streaming

  dmarx

  committed
  e903f04

• cleanup

  dmarx

  committed
  ca4dc8d

• explicit GenerateEncoderDecoderOutput support

  dmarx

  committed
  6e17323

Commits on Mar 7, 2024

• parameterized OutputStreamer tests

  dmarx

  committed
  3e2ff84

• fixed tests

  dmarx

  committed
  256884f

• assert same types on field, cleanup

  dmarx

  committed
  75bf3d7

• tuple-of-tensors output type parity

  dmarx

  committed
  673837b

• cleanup, test type consistency yielded of stream

  dmarx

  committed
  8dd7973

• test emits helpful info on failure

  dmarx

  committed
  cd644fb

• output_attentions streaming for greedy decoding

  dmarx

  committed
  2ad0ead

• fix skipped arguments

  dmarx

  committed
  e523c53

• 'fixed' tests, but now very messy

  dmarx

  committed
  b6c2dc1

• cleaned up

  dmarx

  committed
  7ce491f

Commits on Mar 8, 2024

• test checks all output attrs but past_key_values

  dmarx

  committed
  290b9cb

• test over model varieties

  dmarx

  committed
  41cb5e2

• moved instantiation closer to use

  dmarx

  committed
  a7a8a93

• attention for multinom decoding

  dmarx

  committed
  76ad30d

• contrastive working after multinom supported

  dmarx

  committed
  21adb61

• attention streaming passes all test cases

  dmarx

  committed
  916bf43

• add assistive decoding to test parameterization

  dmarx

  committed
  0b28bab

• draft streaming assisted, exceeds max tokens

  dmarx

  committed
  b89dc83

• back out assisted decoding changes for the moment

  dmarx

  committed
  1e02df6