AssertionError: Input not finite

[dikkeaap]

Hi,

we have recently succesfully trained a model for a plant species sequenced on the MinION using R9.4 flowcell. We have also sequenced the same plant species on the MinION on R10.3 flowcell and scussefully trained a model with those data.

We now have sequenced the same plant (again) on PromethION R10.4 flowcell, but are running into an error when attempting to train a model:

* Taiyaki version 5.1.0
* Platform is Linux-4.15.0-38-generic-x86_64-with-debian-buster-sid
* PyTorch version 1.2.0
* CUDA version 10.0.130 on device GeForce GTX 1080 Ti
* Command line:
* "/opt/kgapps/taiyaki/bin/train_flipflop.py resume2/model_checkpoint_00018.checkpoint mapped_reads_2.hdf5 --min_sub_batch_size 48 --outdir resume3 --lr_max 0.00160 --niteration 40000 --lr_cosine_iters 30000 --overwrite --device 0
* Started on 2020-09-25 08:40:06.741154
* Loading data from mapped_reads_2.hdf5
* Per read file MD5 62e8f6baab6b7ca1d1c046bdaed7e933
* Reads not filtered by id
* Using alphabet definition: canonical alphabet ACGT and no modified bases
* Loaded 14191 reads.
* Reading network from resume2/model_checkpoint_00018.checkpoint
* Network has 10683280 parameters.
* Loaded standard (canonical bases-only) model.
* Dumping initial model
* Sampled 100000 chunks: median(mean_dwell)=9.20, mad(mean_dwell)=0.89
* Learning rate goes like cosine from lr_max to lr_min over 30000.0 iterations.
* At start, train for 200 batches at warm-up learning rate 0.0001
* Standard loss reporting from 141 validation reads held out of training. 
* Standard loss report: chunk length = 5500 & sub-batch size = 48 for 10 sub-batches. 
* Gradient L2 norm cap will be upper 0.05 quantile of the last 100 norms.
* Training
..................................................     1 0.10118 0.10477  116.30s (164.95 ksample/s 18.39 kbase/s) lr=1.00e-04  22.8% chunks filtered
..................................................     2 0.10152 0.10432  116.79s (164.35 ksample/s 18.31 kbase/s) lr=1.00e-04  23.1% chunks filtered
..................................................     3 0.10123 0.10401  110.55s (173.69 ksample/s 19.35 kbase/s) lr=1.00e-04  22.4% chunks filtered
..................................................     4 0.10249 0.10369  117.33s (163.65 ksample/s 18.22 kbase/s) lr=1.00e-04  22.5% chunks filtered
............................Traceback (most recent call last):
  File "/opt/kgapps/taiyaki/bin/train_flipflop.py", line 4, in <module>
    __import__('pkg_resources').run_script('taiyaki==5.1.0', 'train_flipflop.py')
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1446, in run_script
    exec(code, namespace, namespace)
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/taiyaki-5.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/train_flipflop.py", line 624, in <module>
    main()
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/taiyaki-5.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/train_flipflop.py", line 541, in main
    mod_factor_t, calc_grads = True )
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/taiyaki-5.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/train_flipflop.py", line 247, in calculate_loss
    outputs, seqs, seqlens, sharpen)
  File "taiyaki/ctc/ctc.pyx", line 88, in taiyaki.ctc.ctc.FlipFlopCRF.forward
  File "taiyaki/ctc/ctc.pyx", line 62, in taiyaki.ctc.ctc.crf_flipflop_grad
AssertionError: Input not finite

If we resume from the checkpoint, we run into the same error sometime later:

* Taiyaki version 5.1.0
* Platform is Linux-4.15.0-38-generic-x86_64-with-debian-buster-sid
* PyTorch version 1.2.0
* CUDA version 10.0.130 on device GeForce GTX 1080 Ti
* Command line:
* "/opt/kgapps/taiyaki/bin/train_flipflop.py resume2/model_checkpoint_00018.checkpoint mapped_reads_2.hdf5 --min_sub_batch_size 48 --outdir resume3 --lr_max 0.00160 --niteration 40000 --lr_cosine_iters 30000 --overwrite --device 0
* Started on 2020-09-29 08:43:32.712279
* Loading data from mapped_reads_2.hdf5
* Per read file MD5 62e8f6baab6b7ca1d1c046bdaed7e933
* Reads not filtered by id
* Using alphabet definition: canonical alphabet ACGT and no modified bases
* Loaded 14191 reads.
* Reading network from resume2/model_checkpoint_00018.checkpoint
* Network has 10683280 parameters.
* Loaded standard (canonical bases-only) model.
* Dumping initial model
* Sampled 100000 chunks: median(mean_dwell)=9.20, mad(mean_dwell)=0.89
* Learning rate goes like cosine from lr_max to lr_min over 30000.0 iterations.
* At start, train for 200 batches at warm-up learning rate 0.0001
* Standard loss reporting from 141 validation reads held out of training. 
* Standard loss report: chunk length = 5500 & sub-batch size = 48 for 10 sub-batches. 
* Gradient L2 norm cap will be upper 0.05 quantile of the last 100 norms.
* Training
..................................................     1 0.10303 0.09424  114.06s (168.34 ksample/s 18.76 kbase/s) lr=1.00e-04  23.7% chunks filtered
..................................................     2 0.10134 0.09381  115.19s (166.80 ksample/s 18.57 kbase/s) lr=1.00e-04  23.0% chunks filtered
..................................................     3 0.10066 0.09348  121.96s (157.53 ksample/s 17.53 kbase/s) lr=1.00e-04  23.4% chunks filtered
..................................................     4 0.10001 0.09326  115.78s (165.98 ksample/s 18.45 kbase/s) lr=1.00e-04  23.1% chunks filtered
..................................................     5 0.10257 0.09576  112.31s (170.96 ksample/s 19.06 kbase/s) lr=1.60e-03  22.9% chunks filtered
..................................................     6 0.10423 0.09548  112.72s (170.15 ksample/s 18.97 kbase/s) lr=1.60e-03  22.6% chunks filtered
..................................................     7 0.10399 0.09554  116.29s (165.05 ksample/s 18.44 kbase/s) lr=1.60e-03  22.5% chunks filtered
..................................................     8 0.10446 0.09625  115.58s (166.09 ksample/s 18.52 kbase/s) lr=1.60e-03  22.6% chunks filtered
..................................................     9 0.10420 0.09789  115.14s (166.58 ksample/s 18.57 kbase/s) lr=1.60e-03  22.7% chunks filtered
..................................................    10 0.10316 0.09563  111.75s (171.81 ksample/s 19.07 kbase/s) lr=1.60e-03  22.6% chunks filtered
..................................................    11 0.10285 0.09823  113.14s (169.61 ksample/s 18.91 kbase/s) lr=1.60e-03  22.6% chunks filtered
..................................................    12 0.10350 0.09618  113.01s (169.96 ksample/s 18.88 kbase/s) lr=1.60e-03  22.5% chunks filtered
..................................................    13 0.10436 0.09668  118.44s (162.04 ksample/s 18.04 kbase/s) lr=1.60e-03  22.4% chunks filtered
..................................................    14 0.10136 0.09609  120.23s (159.74 ksample/s 17.71 kbase/s) lr=1.60e-03  22.5% chunks filtered
..................................................    15 0.10160 0.09668  114.07s (168.49 ksample/s 18.78 kbase/s) lr=1.60e-03  22.3% chunks filtered
..................................................    16 0.10144 0.09606  121.89s (157.41 ksample/s 17.53 kbase/s) lr=1.60e-03  22.4% chunks filtered
..................................................    17 0.10180 0.09654  113.69s (168.87 ksample/s 18.75 kbase/s) lr=1.60e-03  22.4% chunks filtered
..................................................    18 0.10381 0.09578  114.91s (166.92 ksample/s 18.63 kbase/s) lr=1.60e-03  22.3% chunks filtered
..................................................    19 0.10369 0.09621  119.26s (161.01 ksample/s 17.88 kbase/s) lr=1.60e-03  22.4% chunks filtered
.................................................C    20 0.10542 0.09585  116.87s (164.28 ksample/s 18.32 kbase/s) lr=1.60e-03  22.4% chunks filtered
..................................................    21 0.10229 0.09613  112.86s (170.12 ksample/s 18.89 kbase/s) lr=1.60e-03  22.4% chunks filtered
..................................................    22 0.10319 0.09585  119.18s (161.01 ksample/s 17.88 kbase/s) lr=1.60e-03  22.4% chunks filtered
..................................................    23 0.10428 0.09592  112.53s (170.74 ksample/s 19.03 kbase/s) lr=1.60e-03  22.3% chunks filtered
..................................................    24 0.10430 0.09659  118.36s (162.13 ksample/s 18.06 kbase/s) lr=1.60e-03  22.3% chunks filtered
..................................................    25 0.10297 0.09839  114.13s (168.21 ksample/s 18.72 kbase/s) lr=1.60e-03  22.3% chunks filtered
..................................................    26 0.10191 0.09591  119.56s (160.64 ksample/s 17.88 kbase/s) lr=1.60e-03  22.4% chunks filtered
..................................................    27 0.10201 0.09587  119.00s (161.43 ksample/s 18.00 kbase/s) lr=1.59e-03  22.4% chunks filtered
..................................................    28 0.10112 0.09673  115.11s (166.81 ksample/s 18.54 kbase/s) lr=1.59e-03  22.3% chunks filtered
..................................................    29 0.10262 0.09598  116.90s (164.09 ksample/s 18.27 kbase/s) lr=1.59e-03  22.3% chunks filtered
..................................................    30 0.10112 0.09629  119.62s (160.46 ksample/s 17.91 kbase/s) lr=1.59e-03  22.4% chunks filtered
..................................................    31 0.10133 0.09634  113.04s (169.94 ksample/s 18.97 kbase/s) lr=1.59e-03  22.4% chunks filtered
..................................................    32 0.10289 0.09638  114.80s (167.16 ksample/s 18.62 kbase/s) lr=1.59e-03  22.4% chunks filtered
..................................................    33 0.10126 0.09713  118.53s (162.10 ksample/s 18.09 kbase/s) lr=1.59e-03  22.4% chunks filtered
..................................................    34 0.10132 0.09618  114.32s (167.75 ksample/s 18.63 kbase/s) lr=1.59e-03  22.4% chunks filtered
..................................................    35 0.10256 0.09735  114.83s (167.14 ksample/s 18.61 kbase/s) lr=1.59e-03  22.4% chunks filtered
..................................................    36 0.10273 0.09627  118.25s (162.45 ksample/s 18.04 kbase/s) lr=1.59e-03  22.5% chunks filtered
..................................................    37 0.10070 0.09690  119.18s (160.95 ksample/s 17.92 kbase/s) lr=1.59e-03  22.5% chunks filtered
..................................................    38 0.10240 0.09587  122.12s (157.21 ksample/s 17.53 kbase/s) lr=1.59e-03  22.5% chunks filtered
..................................................    39 0.10228 0.09652  121.98s (157.32 ksample/s 17.50 kbase/s) lr=1.59e-03  22.5% chunks filtered
.................................................C    40 0.10154 0.09753  119.72s (160.40 ksample/s 17.86 kbase/s) lr=1.59e-03  22.5% chunks filtered
..................................................    41 0.10145 0.09766  116.83s (164.52 ksample/s 18.34 kbase/s) lr=1.59e-03  22.5% chunks filtered
..................................................    42 0.10305 0.09752  116.33s (165.06 ksample/s 18.37 kbase/s) lr=1.59e-03  22.5% chunks filtered
..................................................    43 0.10309 0.09718  113.95s (168.32 ksample/s 18.77 kbase/s) lr=1.58e-03  22.5% chunks filtered
..................................................    44 0.10519 0.09719  117.84s (163.13 ksample/s 18.14 kbase/s) lr=1.58e-03  22.5% chunks filtered
..................................................    45 0.10280 0.09720  114.28s (167.86 ksample/s 18.66 kbase/s) lr=1.58e-03  22.5% chunks filtered
..................................................    46 0.10338 0.09686  117.43s (163.33 ksample/s 18.20 kbase/s) lr=1.58e-03  22.5% chunks filtered
..................................................    47 0.10158 0.09750  117.49s (163.25 ksample/s 18.12 kbase/s) lr=1.58e-03  22.5% chunks filtered
..................................................    48 0.10328 0.09947  117.06s (163.95 ksample/s 18.26 kbase/s) lr=1.58e-03  22.5% chunks filtered
..................................................    49 0.10487 0.09910  115.14s (166.81 ksample/s 18.65 kbase/s) lr=1.58e-03  22.6% chunks filtered
..................................................    50 0.10118 0.09751  120.68s (159.18 ksample/s 17.63 kbase/s) lr=1.58e-03  22.6% chunks filtered
..................................................    51 0.10375 0.09986  115.22s (166.70 ksample/s 18.58 kbase/s) lr=1.58e-03  22.6% chunks filtered
..................................................    52 0.10603 0.09855  119.27s (161.08 ksample/s 17.95 kbase/s) lr=1.58e-03  22.6% chunks filtered
..................................................    53 0.10155 0.09740  116.16s (165.29 ksample/s 18.38 kbase/s) lr=1.58e-03  22.6% chunks filtered
..................................................    54 0.10260 0.09768  112.45s (170.72 ksample/s 18.98 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    55 0.10155 0.09779  116.01s (165.47 ksample/s 18.41 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    56 0.10258 0.09721  117.70s (163.23 ksample/s 18.10 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    57 0.10468 0.09874  118.00s (162.63 ksample/s 18.08 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    58 0.10292 0.09673  120.48s (159.47 ksample/s 17.73 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    59 0.10120 0.09683  116.46s (164.98 ksample/s 18.34 kbase/s) lr=1.57e-03  22.6% chunks filtered
.................................................C    60 0.10291 0.09715  111.07s (172.95 ksample/s 19.24 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    61 0.10265 0.09764  117.70s (163.35 ksample/s 18.17 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    62 0.10124 0.09724  118.02s (162.72 ksample/s 18.12 kbase/s) lr=1.57e-03  22.6% chunks filtered
..................................................    63 0.10444 0.09788  116.20s (165.41 ksample/s 18.37 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    64 0.10290 0.09740  115.10s (166.85 ksample/s 18.55 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    65 0.10396 0.09741  116.88s (164.25 ksample/s 18.32 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    66 0.10418 0.09737  120.62s (159.13 ksample/s 17.71 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    67 0.10352 0.09803  113.83s (168.62 ksample/s 18.76 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    68 0.10091 0.09806  114.76s (167.36 ksample/s 18.64 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    69 0.10277 0.09752  109.89s (174.39 ksample/s 19.44 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    70 0.10152 0.09761  121.69s (157.95 ksample/s 17.54 kbase/s) lr=1.56e-03  22.6% chunks filtered
..................................................    71 0.10293 0.09910  112.92s (169.99 ksample/s 18.91 kbase/s) lr=1.55e-03  22.7% chunks filtered
..................................................    72 0.10342 0.09704  115.96s (165.39 ksample/s 18.41 kbase/s) lr=1.55e-03  22.7% chunks filtered
..................................................    73 0.10179 0.09838  112.01s (171.42 ksample/s 19.05 kbase/s) lr=1.55e-03  22.6% chunks filtered
..................................................    74 0.10373 0.09874  111.87s (171.61 ksample/s 19.09 kbase/s) lr=1.55e-03  22.6% chunks filtered
..................................................    75 0.10389 0.09729  110.51s (173.63 ksample/s 19.31 kbase/s) lr=1.55e-03  22.6% chunks filtered
..................................................    76 0.10115 0.09802  116.14s (165.44 ksample/s 18.41 kbase/s) lr=1.55e-03  22.6% chunks filtered
..................................................    77 0.10189 0.09826  121.15s (158.53 ksample/s 17.68 kbase/s) lr=1.55e-03  22.6% chunks filtered
..................................................    78 0.10195 0.09802  115.58s (165.89 ksample/s 18.42 kbase/s) lr=1.54e-03  22.6% chunks filtered
..................................................    79 0.10193 0.09749  120.47s (159.53 ksample/s 17.80 kbase/s) lr=1.54e-03  22.6% chunks filtered
.................................................C    80 0.10151 0.09734  121.18s (158.53 ksample/s 17.65 kbase/s) lr=1.54e-03  22.6% chunks filtered
..................................................    81 0.10123 0.09833  114.94s (166.94 ksample/s 18.59 kbase/s) lr=1.54e-03  22.6% chunks filtered
..................................................    82 0.10002 0.09774  122.28s (157.09 ksample/s 17.51 kbase/s) lr=1.54e-03  22.6% chunks filtered
..................................................    83 0.10148 0.09776  120.81s (158.98 ksample/s 17.69 kbase/s) lr=1.54e-03  22.6% chunks filtered
..................................................    84 0.10369 0.09914  118.36s (162.27 ksample/s 18.08 kbase/s) lr=1.54e-03  22.7% chunks filtered
..................................................    85 0.10256 0.09831  118.93s (161.22 ksample/s 17.93 kbase/s) lr=1.53e-03  22.7% chunks filtered
..................................................    86 0.10094 0.09856  117.98s (162.84 ksample/s 18.06 kbase/s) lr=1.53e-03  22.7% chunks filtered
..................................................    87 0.10262 0.09856  115.67s (165.98 ksample/s 18.48 kbase/s) lr=1.53e-03  22.7% chunks filtered
..................................................    88 0.10268 0.09754  114.59s (167.54 ksample/s 18.67 kbase/s) lr=1.53e-03  22.6% chunks filtered
..................................................    89 0.10122 0.09729  115.43s (166.36 ksample/s 18.49 kbase/s) lr=1.53e-03  22.6% chunks filtered
..................................................    90 0.10119 0.09826  117.85s (162.79 ksample/s 18.10 kbase/s) lr=1.53e-03  22.6% chunks filtered
..................................................    91 0.10304 0.09846  113.75s (168.66 ksample/s 18.74 kbase/s) lr=1.52e-03  22.6% chunks filtered
..................................................    92 0.10238 0.09824  112.37s (171.00 ksample/s 19.04 kbase/s) lr=1.52e-03  22.6% chunks filtered
..................................................    93 0.10246 0.09978  113.78s (168.81 ksample/s 18.80 kbase/s) lr=1.52e-03  22.6% chunks filtered
..................................................    94 0.10248 0.09794  116.51s (164.81 ksample/s 18.42 kbase/s) lr=1.52e-03  22.6% chunks filtered
..................................................    95 0.10194 0.09909  116.15s (165.32 ksample/s 18.43 kbase/s) lr=1.52e-03  22.6% chunks filtered
..................................................    96 0.10665 0.10063  112.00s (171.39 ksample/s 19.09 kbase/s) lr=1.51e-03  22.6% chunks filtered
..................................................    97 0.10250 0.09883  119.84s (160.29 ksample/s 17.83 kbase/s) lr=1.51e-03  22.6% chunks filtered
..................................................    98 0.10183 0.09980  121.37s (158.07 ksample/s 17.59 kbase/s) lr=1.51e-03  22.6% chunks filtered
..................................................    99 0.10106 0.09903  123.88s (155.04 ksample/s 17.33 kbase/s) lr=1.51e-03  22.6% chunks filtered
.................................................C   100 0.10287 0.09871  116.61s (164.78 ksample/s 18.32 kbase/s) lr=1.51e-03  22.6% chunks filtered
..................................................   101 0.09959 0.09830  115.93s (165.56 ksample/s 18.41 kbase/s) lr=1.51e-03  22.6% chunks filtered
..................................................   102 0.10232 0.10040  116.30s (165.17 ksample/s 18.37 kbase/s) lr=1.50e-03  22.6% chunks filtered
..................................................   103 0.10204 0.09922  113.61s (168.76 ksample/s 18.80 kbase/s) lr=1.50e-03  22.6% chunks filtered
..................................................   104 0.10118 0.10029  115.30s (166.43 ksample/s 18.55 kbase/s) lr=1.50e-03  22.6% chunks filtered
..................................................   105 0.10316 0.09928  119.12s (161.12 ksample/s 17.96 kbase/s) lr=1.50e-03  22.7% chunks filtered
..................................................   106 0.10159 0.09964  114.67s (167.49 ksample/s 18.65 kbase/s) lr=1.50e-03  22.6% chunks filtered
..................................................   107 0.10175 0.09931  115.64s (165.94 ksample/s 18.48 kbase/s) lr=1.49e-03  22.6% chunks filtered
..................................................   108 0.10043 0.09866  125.75s (152.72 ksample/s 17.02 kbase/s) lr=1.49e-03  22.7% chunks filtered
..................................................   109 0.10020 0.09867  118.72s (161.77 ksample/s 18.02 kbase/s) lr=1.49e-03  22.7% chunks filtered
..................................................   110 0.10340 0.09991  116.44s (164.87 ksample/s 18.40 kbase/s) lr=1.49e-03  22.7% chunks filtered
..................................................   111 0.10187 0.09888  120.00s (160.10 ksample/s 17.83 kbase/s) lr=1.49e-03  22.7% chunks filtered
..................................................   112 0.10131 0.09886  116.21s (165.18 ksample/s 18.35 kbase/s) lr=1.48e-03  22.7% chunks filtered
..................................................   113 0.10127 0.09878  119.95s (159.88 ksample/s 17.79 kbase/s) lr=1.48e-03  22.7% chunks filtered
..................................................   114 0.10092 0.09867  116.38s (164.89 ksample/s 18.39 kbase/s) lr=1.48e-03  22.7% chunks filtered
..................................................   115 0.10235 0.09870  117.45s (163.42 ksample/s 18.19 kbase/s) lr=1.48e-03  22.7% chunks filtered
..................................................   116 0.10141 0.09837  118.39s (162.21 ksample/s 18.07 kbase/s) lr=1.47e-03  22.7% chunks filtered
..................................................   117 0.10130 0.09781  121.45s (158.15 ksample/s 17.60 kbase/s) lr=1.47e-03  22.7% chunks filtered
..................................................   118 0.09917 0.09845  121.79s (157.63 ksample/s 17.54 kbase/s) lr=1.47e-03  22.7% chunks filtered
..................................................   119 0.10108 0.09849  118.65s (161.84 ksample/s 17.97 kbase/s) lr=1.47e-03  22.7% chunks filtered
.................................................C   120 0.09964 0.09825  121.11s (158.32 ksample/s 17.58 kbase/s) lr=1.47e-03  22.7% chunks filtered
..................................................   121 0.10007 0.09785  115.92s (165.46 ksample/s 18.38 kbase/s) lr=1.46e-03  22.7% chunks filtered
..................................................   122 0.09985 0.09845  114.33s (168.07 ksample/s 18.72 kbase/s) lr=1.46e-03  22.7% chunks filtered
..................................................   123 0.09904 0.09813  120.89s (158.90 ksample/s 17.63 kbase/s) lr=1.46e-03  22.7% chunks filtered
..................................................   124 0.10266 0.09908  117.09s (163.89 ksample/s 18.23 kbase/s) lr=1.46e-03  22.7% chunks filtered
..................................................   125 0.10281 0.09849  118.34s (162.25 ksample/s 18.06 kbase/s) lr=1.45e-03  22.7% chunks filtered
..................................................   126 0.10141 0.09834  116.42s (164.98 ksample/s 18.35 kbase/s) lr=1.45e-03  22.7% chunks filtered
..................................................   127 0.09954 0.09840  117.82s (162.89 ksample/s 18.10 kbase/s) lr=1.45e-03  22.7% chunks filtered
..................................................   128 0.09923 0.09771  115.27s (166.61 ksample/s 18.57 kbase/s) lr=1.45e-03  22.7% chunks filtered
..................................................   129 0.10268 0.09790  110.59s (173.61 ksample/s 19.28 kbase/s) lr=1.45e-03  22.7% chunks filtered
..................................................   130 0.09932 0.09760  115.02s (166.87 ksample/s 18.58 kbase/s) lr=1.44e-03  22.7% chunks filtered
..................................................   131 0.10253 0.09847  113.70s (168.84 ksample/s 18.83 kbase/s) lr=1.44e-03  22.7% chunks filtered
..................................................   132 0.09926 0.09810  114.94s (166.99 ksample/s 18.55 kbase/s) lr=1.44e-03  22.7% chunks filtered
..................................................   133 0.10006 0.09842  120.04s (160.03 ksample/s 17.81 kbase/s) lr=1.44e-03  22.7% chunks filtered
..................................................   134 0.10196 0.09928  115.63s (166.08 ksample/s 18.52 kbase/s) lr=1.43e-03  22.7% chunks filtered
..................................................   135 0.09793 0.09739  119.16s (161.16 ksample/s 17.89 kbase/s) lr=1.43e-03  22.7% chunks filtered
..................................................   136 0.10120 0.09817  113.37s (169.33 ksample/s 18.89 kbase/s) lr=1.43e-03  22.7% chunks filtered
..................................................   137 0.10041 0.09765  110.80s (173.22 ksample/s 19.23 kbase/s) lr=1.43e-03  22.7% chunks filtered
......Traceback (most recent call last):
  File "/opt/kgapps/taiyaki/bin/train_flipflop.py", line 4, in <module>
    __import__('pkg_resources').run_script('taiyaki==5.1.0', 'train_flipflop.py')
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1446, in run_script
    exec(code, namespace, namespace)
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/taiyaki-5.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/train_flipflop.py", line 624, in <module>
    main()
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/taiyaki-5.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/train_flipflop.py", line 541, in main
    mod_factor_t, calc_grads = True )
  File "/opt/kgapps/taiyaki/lib/python3.7/site-packages/taiyaki-5.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/train_flipflop.py", line 247, in calculate_loss
    outputs, seqs, seqlens, sharpen)
  File "taiyaki/ctc/ctc.pyx", line 88, in taiyaki.ctc.ctc.FlipFlopCRF.forward
  File "taiyaki/ctc/ctc.pyx", line 62, in taiyaki.ctc.ctc.crf_flipflop_grad
AssertionError: Input not finite

Do you have any idea what is going on here, and what we are doing wrong?

[marcus1487]

This is an area of active research internally. Currently the best solution/workaround is to decrease the --max_lr and increase the --niterations (and maybe --lr_cosine_iters).

[SCDealy]

I just generated the same exception using the example training set:

  https://s3-eu-west-1.amazonaws.com/ont-research/taiyaki_walkthrough.tar.gz

while following the taiyaki walk through instructions:

  https://github.com/nanoporetech/taiyaki/blob/master/docs/walkthrough.rst

though it took 556 iterations before it failed.

FWIW.