Code

Weights Metrics#340

Code

Open

ElliotStein wants to merge 67 commits intoarcee-ai:mainarcee-ai/mergekit:mainfrom ElliotStein:weights_metricsElliotStein/mergekit:weights_metricsCopy head branch name to clipboard

+3,354-31

Commits

Commits on May 30, 2024

• introduced metric_methods, closely following implementation of merge_methods

  

  [email protected]

  committed
  c2b0f06

• measure.py closely follows metric.py to apply chosen metrics

  

  [email protected]

  committed
  64e54bf

• plot tools include class to handle output of run_metrics, and a class to plot (simple) interactive graph

  

  [email protected]

  committed
  7e37d69

• only minor changes required to existing mergekit code

  

  [email protected]

  committed
  2effd41

• only minor changes required to existing mergekit

  

  [email protected]

  committed
  ed59d9b

Commits on Jun 3, 2024

• Implemented interactive dashboard for metrics visualisation

  

  [email protected]

  committed
  564d45c

• Remove single-model stats for now. Bring MSE into all_metrics

  

  [email protected]

  committed
  e530568

Commits on Jun 7, 2024

• Introduce attention weights and restructure dashboard

  

  [email protected]

  committed
  207e874

Commits on Jun 10, 2024

• refine implementation of attention metrics, add line plots to dashboard

  

  [email protected]

  committed
  b30175e

Commits on Jun 11, 2024

• More restructuring, more seamless integration of attention and mlp layers. heatmaps for other metrics.

  

  [email protected]

  committed
  dfc0603

Commits on Jun 12, 2024

• vectorise heatmap computation

  

  [email protected]

  committed
  f88f904

Commits on Jun 14, 2024

• Address issue with lexicographical sort by adding leading zeros to layer name

  

  [email protected]

  committed
  f89df57

• remove unnecessary import from last commit

  

  [email protected]

  committed
  74c5d33

Commits on Jun 17, 2024

• add validation check to ensure MergeConfiguration method is either Merge OR Metri, not both.

  

  [email protected]

  committed
  7c209d2

• moved run_metrics and use click to enable commandline control over arguments

  

  [email protected]

  committed
  db65f83

• rename example config

  

  [email protected]

  committed
  3fe66a8

• correct case for gqa_group name

  

  [email protected]

  committed
  62692d1

• replace measure with merge + early out

  

  [email protected]

  committed
  2a0c520

• guard against divide by zero

  

  [email protected]

  committed
  f71e360

• restore plan_to_disk functionality for merging. Move metrics planning to separate case

  

  [email protected]

  committed
  7e14266

• add optional interactive plot packages

  

  [email protected]

  committed
  0f3430f

• minor cleanup

  

  [email protected]

  committed
  8d68e39

• Merge remote-tracking branch 'upstream/main'

  

  [email protected]

  committed
  59e23fe

• Merge branch 'main' into weights_metrics

  

  [email protected]

  committed
  89ecbf5

Commits on Jun 19, 2024

• Add GQA info to (llama) architecture and refactor

  

  [email protected]

  committed
  404e395

• Pass GQA info from architecture json all the way to attn metrics. Generalised substitute function in architecture

  

  [email protected]

  committed
  8e3c861

• re-organised and simplified dashboard view

  

  [email protected]

  committed
  a0e8c27

• colour-categorise lineplot points by layertime

  

  [email protected]

  committed
  7e2b552

Commits on Jun 24, 2024

• restructure and refactor results and plotting

  

  [email protected]

  committed
  69c3b15

Commits on Jun 26, 2024

• restructure metrics storage, remove graph from plot, remove redundancies and refactor

  

  [email protected]

  committed
  f36fd70

• Add intra-layer metrics, completed implementation of changes from prev commit

  

  [email protected]

  committed
  50a5716

Commits on Jul 4, 2024

• restructure metrics for modularity

  ElliotStein

  committed
  73dd3ae

• add load and save functions to base

  ElliotStein

  committed
  ddbb475

• Internal Representaions analysis - first commit

  ElliotStein

  committed
  36d28b0

Commits on Jul 8, 2024

• refactor variable name for consistency

  ElliotStein

  committed
  fa6d098

• Plot heatmaps stored in results.others

  ElliotStein

  committed
  4949458

• generalised results handler to load from metrics list or ready-made Results object

  ElliotStein

  committed
  bd62ed9

Commits on Jul 9, 2024

• abstracted and add skip block analysis

  ElliotStein

  committed
  9c87514

• refactor and restructure

  ElliotStein

  committed
  4ce67b0

• clean up imports and remove hard coding

  ElliotStein

  committed
  88ed18e

• tidy up tqdm

  ElliotStein

  committed
  1041298

• improve robustness of load and save using pathlib

  ElliotStein

  committed
  01d5a2b

• reintroduced heatmap functionality

  ElliotStein

  committed
  f64a71e

• allow for plot keyworks to be passed into Heatmap object

  ElliotStein

  committed
  8b05c2a

• example config

  ElliotStein

  committed
  b8f7e32

• improve implementation consistency

  ElliotStein

  committed
  c4df572

Commits on Jul 16, 2024

• experimental linearity score metric

  ElliotStein

  committed
  5e051a8

• add matplotlib to optional dependencies

  ElliotStein

  committed
  bb8fce2

• Merge remote-tracking branch 'upstream/main' into weights_metrics

  ElliotStein

  committed
  9c92efd

• remove quantisation and update environment reqs

  ElliotStein

  committed
  e1c1ecd

Commits on Jul 22, 2024

• tidy up and add missing dependency

  ElliotStein

  committed
  9db9d3b

Commits on Jul 25, 2024

• Major restructuring of Results, Results handling, metrics

  ElliotStein

  committed
  af2bf1a

Commits on Jul 30, 2024

• address alphanumeric representation layer name ordering issue

  ElliotStein

  committed
  6045980

• MAJOR RESTRUCTURE of results, results handler and representation metrics

  ElliotStein

  committed
  cb5e31a

• tidy up

  ElliotStein

  committed
  dfdcb00

Commits on Jul 31, 2024

• further restructuring

  ElliotStein

  committed
  7bc8001

Commits on Aug 1, 2024

• further refinements to representations experiment

  ElliotStein

  committed
  a5c5811

• added necessary imports and modules

  ElliotStein

  committed
  3d3fd33

• minor restructuring and refactoring

  ElliotStein

  committed
  2dfa4f9

• Bug fixes

  ElliotStein

  committed
  60726d4

Commits on Aug 2, 2024

• visualisation fixes

  ElliotStein

  committed
  891bbb4

• address model path naming issue (but only for representations! Not model weights)

  ElliotStein

  committed
  04b64e8

Commits on Aug 5, 2024

• minor fix

  ElliotStein

  committed
  1b5a3c0

Commits on Aug 6, 2024

• More fixes, end-to-end tested and working

  ElliotStein

  committed
  6bd12e7

• refactor folder name

  ElliotStein

  committed
  5ac4c78

Commits on Aug 7, 2024

• Implement CKNNA and PCA visualisation

  ElliotStein

  committed
  68a6a30

• fixes to CKNNA and tidy up

  ElliotStein

  committed
  20dd0f6