Is there a code to generate data for music-dissector?

[vpavlenko]

Hi,

Thank you so much for putting together both All-in-One and Music-Dissector. Music-Dissector looks fantastic, and I'd like to open my own files in it. I've found about it when triaging an issue with Cython and madmom to build WaveBeat, as I prepare a doc on the current state of transcription.

I have successfully run allin1 locally to get the data:

I see that music-dissector loads data in a different form to show three energy spectra: {"nav": {"bass": {"low": [2, 2, 2, 2, 2 etc.

Is there a code to generate data from demucs and All-in-One results into this music-dissector format?

[tae-jun]

Hi, I'm glad that you could successfully install and run allin1 🥹
And I'm surprised that you are trying to open your own files on Music Dissector!

Well, I don't have a plan to release the code generating input data for Music Dissector officially.
However, I can give you the code snippets.

I guess what's missing is the code for generating waveforms, right?
As you may already noticed, it's 3-band waveforms: blue=low, yellow=vocal, white=high.
I simply extracted those energies from spectrograms which just allin1 uses.
specs are just the same spectrograms here in allin1.

LOW = 250
HIGH = 4000
FPS = 100

BIN_FREQS = [
  43.06640625, 64.599609375, 86.1328125, 107.666015625, 129.19921875, 150.732421875, 172.265625, 193.798828125,
  215.33203125, 236.865234375, 258.3984375, 279.931640625, 301.46484375, 322.998046875, 344.53125, 366.064453125,
  387.59765625, 409.130859375, 430.6640625, 452.197265625, 495.263671875, 516.796875, 538.330078125, 581.396484375,
  624.462890625, 645.99609375, 689.0625, 732.12890625, 775.1953125, 839.794921875, 882.861328125, 925.927734375,
  990.52734375, 1055.126953125, 1098.193359375, 1184.326171875, 1248.92578125, 1313.525390625, 1399.658203125,
  1485.791015625, 1571.923828125, 1658.056640625, 1765.72265625, 1873.388671875, 1981.0546875, 2088.720703125,
  2217.919921875, 2347.119140625, 2497.8515625, 2627.05078125, 2799.31640625, 2950.048828125, 3143.84765625,
  3316.11328125, 3509.912109375, 3725.244140625, 3940.576171875, 4177.44140625, 4435.83984375, 4694.23828125,
  4974.169921875, 5275.634765625, 5577.099609375, 5921.630859375, 6266.162109375, 6653.759765625, 7041.357421875,
  7450.48828125, 7902.685546875, 8376.416015625, 8871.6796875, 9388.4765625, 9948.33984375, 10551.26953125,
  11175.732421875, 11843.26171875, 12553.857421875, 13285.986328125, 14082.71484375, 14922.509765625, 15805.37109375
]
BIN_FREQS = np.array(BIN_FREQS).round().astype(int)


def process(ex, outdir: Path):
  specs = ex['spec']
  
  i_low = np.flatnonzero(BIN_FREQS < LOW)
  i_high = np.flatnonzero(BIN_FREQS > HIGH)
  i_mid = np.flatnonzero((LOW <= BIN_FREQS) & (BIN_FREQS <= HIGH))
  
  # Compute the max energy value for each frequency band considering all instruments.
  max_low = specs[:, :, i_low].max()
  max_mid = specs[:, :, i_mid].max()
  max_high = specs[:, :, i_high].max()
  
  wavs_low, wavs_mid, wavs_high = [
    specs[:, :, indices].mean(axis=-1)
    # spec[:, indices].mean(axis=1)
    for indices in [i_low, i_mid, i_high]
  ]
  wavs_low /= max_low
  wavs_mid /= max_mid
  wavs_high /= max_high
  assert wavs_low.max() <= 1.0
  assert wavs_mid.max() <= 1.0
  assert wavs_high.max() <= 1.0
  
  navs_low = np.array([median_filter(wav, size=FPS) for wav in wavs_low])
  navs_mid = np.array([median_filter(wav, size=FPS) for wav in wavs_mid])
  navs_high = np.array([median_filter(wav, size=FPS) for wav in wavs_high])
  
  navs_low = navs_low
  navs_mid = navs_low + navs_mid
  navs_high = navs_mid + navs_high
  
  max_nav = np.max([navs_low.max(), navs_mid.max(), navs_high.max()])
  navs_low /= max_nav
  navs_mid /= max_nav
  navs_high /= max_nav
  assert navs_high.max() <= 1.0
  
  data = {
    'nav': {},
    'wav': {},
  }
  
  for (
    eg_low, eg_mid, eg_high,
    nav_low, nav_mid, nav_high,
    inst
  ) in zip(
    wavs_low, wavs_mid, wavs_high,
    navs_low, navs_mid, navs_high,
    [
      'bass',
      'drum',
      'other',
      'vocal',
    ]
  ):
    data['wav'][inst] = {
      'low': eg_low,
      'mid': eg_mid,
      'high': eg_high,
    }
    
    data['nav'][inst] = {
      'low': nav_low,
      'mid': nav_mid,
      'high': nav_high,
    }
  
  apply_to_dict(data, to_uint8_list)
  data['duration'] = specs.shape[1] / FPS
  
  outpath = outdir / f'{ex["track_key"]}.json'
  outpath.write_text(json.dumps(data))

I hope this is helpful for you!