Skip to content

Commit

Permalink
Merge pull request espnet#5744 from cromz22/iwslt24_indic
Browse files Browse the repository at this point in the history
Add recipe for IWSLT 2024 shared task Indic track
  • Loading branch information
sw005320 authored May 8, 2024
2 parents 89dcf8b + 5180c85 commit 435803c
Show file tree
Hide file tree
Showing 23 changed files with 755 additions and 0 deletions.
1 change: 1 addition & 0 deletions egs2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ See: https://espnet.github.io/espnet/espnet2_tutorial.html#recipes-using-espnet2
| iwslt21_low_resource | ALFFA, IARPA Babel, Gamayun, IWSLT 2021 | ASR | SWA | http://www.openslr.org/25/ https://catalog.ldc.upenn.edu/LDC2017S05 https://gamayun.translatorswb.org/data/ https://iwslt.org/2021/low-resource | |
| iwslt22_dialect | IWSLT2022 dialectal speech translation shared task | ASR/ST | ARA->Tunisian ARA | https://github.com/kevinduh/iwslt22-dialect.git | |
| iwslt22_low_resource | IWSLT2022 Low-resource speech translation track task | ST | Tamasheq->FrenchPermalink | https://github.com/mzboito/IWSLT2022_Tamasheq_data.git |
| iwslt24_indic | IWSLT2024 Indic speech translation track | ST | ENG -> HIN, BEN, TAM | https://iwslt.org/2024/indic | |
| jdcinal | Japanese Dialogue Corpus of Information Navigation and Attentive Listening Annotated with Extended ISO-24617-2 Dialogue Act Tags | SLU | JPN | http://www.lrec-conf.org/proceedings/lrec2018/pdf/464.pdf http://tts.speech.cs.cmu.edu/awb/infomation_navigation_and_attentive_listening_0.2.zip | |
| jkac | J-KAC: Japanese Kamishibai and audiobook corpus | TTS | JPN | https://sites.google.com/site/shinnosuketakamichi/research-topics/j-kac_corpus | |
| jmd | JMD: Japanese multi-dialect corpus for speech synthesis | TTS | JPN | https://sites.google.com/site/shinnosuketakamichi/research-topics/jmd_corpus | |
Expand Down
1 change: 1 addition & 0 deletions egs2/TEMPLATE/asr1/db.sh
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ CMU_INDIC=downloads
INDIC_SPEECH=downloads
IWSLT22_DIALECT=
IWSLT22_LOW_RESOURCE=downloads
IWSLT24_INDIC=
JKAC=
MUCS_SUBTASK1=downloads
MUCS_SUBTASK2=downloads
Expand Down
67 changes: 67 additions & 0 deletions egs2/iwslt24_indic/st1/RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# RESULTS

## En-Hi

### Environments
- date: `Thu Apr 18 01:34:53 JST 2024`
- python version: `3.10.14 (main, Mar 21 2024, 16:24:04) [GCC 11.2.0]`
- espnet version: `espnet 202402`
- pytorch version: `pytorch 2.1.0`
- Git hash: `83c179ab842987cf01642df2db372aaae260df55`
- Commit date: `Wed Apr 17 00:28:29 2024 +0900`

### Model config

- training: [./conf/tuning/train_st_conformer.yaml](./conf/tuning/train_st_conformer.yaml)
- decoding: [./conf/tuning/decode_st_conformer.yaml](./conf/tuning/decode_st_conformer.yaml)
- model url: [https://huggingface.co/espnet/iwslt24_indic_en_hi_bpe_tc4000](https://huggingface.co/espnet/iwslt24_indic_en_hi_bpe_tc4000)

### BLEU

|dataset|score|verbose_score|
|---|---|---|
|decode_st_conformer_st_model_valid.acc.ave/dev.en-hi|37.1|64.8/44.9/34.2/26.2 (BP = 0.924 ratio = 0.927 hyp_len = 195297 ref_len = 210636)|

## En-Bn

### Environments
- date: `Wed Apr 17 02:51:38 JST 2024`
- python version: `3.10.14 (main, Mar 21 2024, 16:24:04) [GCC 11.2.0]`
- espnet version: `espnet 202402`
- pytorch version: `pytorch 2.1.0`
- Git hash: `83c179ab842987cf01642df2db372aaae260df55`
- Commit date: `Wed Apr 17 00:28:29 2024 +0900`

### Model config

- training: [./conf/tuning/train_st_conformer.yaml](./conf/tuning/train_st_conformer.yaml)
- decoding: [./conf/tuning/decode_st_conformer.yaml](./conf/tuning/decode_st_conformer.yaml)
- model url: [https://huggingface.co/espnet/iwslt24_indic_en_bn_bpe_tc4000](https://huggingface.co/espnet/iwslt24_indic_en_bn_bpe_tc4000)

### BLEU

|dataset|score|verbose_score|
|---|---|---|
|decode_st_conformer_st_model_valid.acc.ave/dev.en-bn|2.1|19.7/3.6/1.0/0.3 (BP = 1.000 ratio = 1.185 hyp_len = 46094 ref_len = 38883)|

# En-Ta

## Environments
- date: `Thu Apr 18 01:03:59 JST 2024`
- python version: `3.10.14 (main, Mar 21 2024, 16:24:04) [GCC 11.2.0]`
- espnet version: `espnet 202402`
- pytorch version: `pytorch 2.1.0`
- Git hash: `83c179ab842987cf01642df2db372aaae260df55`
- Commit date: `Wed Apr 17 00:28:29 2024 +0900`

### Model config

- training: [./conf/tuning/train_st_conformer.yaml](./conf/tuning/train_st_conformer.yaml)
- decoding: [./conf/tuning/decode_st_conformer.yaml](./conf/tuning/decode_st_conformer.yaml)
- model url: [https://huggingface.co/espnet/iwslt24_indic_en_ta_bpe_tc4000](https://huggingface.co/espnet/iwslt24_indic_en_ta_bpe_tc4000)

### BLEU

|dataset|score|verbose_score|
|---|---|---|
|decode_st_conformer_st_model_valid.acc.ave/dev.en-ta|6.3|46.5/9.4/4.7/1.9 (BP = 0.798 ratio = 0.816 hyp_len = 66168 ref_len = 81059)|
110 changes: 110 additions & 0 deletions egs2/iwslt24_indic/st1/cmd.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# ====== About run.pl, queue.pl, slurm.pl, and ssh.pl ======
# Usage: <cmd>.pl [options] JOB=1:<nj> <log> <command...>
# e.g.
# run.pl --mem 4G JOB=1:10 echo.JOB.log echo JOB
#
# Options:
# --time <time>: Limit the maximum time to execute.
# --mem <mem>: Limit the maximum memory usage.
# -–max-jobs-run <njob>: Limit the number parallel jobs. This is ignored for non-array jobs.
# --num-threads <ngpu>: Specify the number of CPU core.
# --gpu <ngpu>: Specify the number of GPU devices.
# --config: Change the configuration file from default.
#
# "JOB=1:10" is used for "array jobs" and it can control the number of parallel jobs.
# The left string of "=", i.e. "JOB", is replaced by <N>(Nth job) in the command and the log file name,
# e.g. "echo JOB" is changed to "echo 3" for the 3rd job and "echo 8" for 8th job respectively.
# Note that the number must start with a positive number, so you can't use "JOB=0:10" for example.
#
# run.pl, queue.pl, slurm.pl, and ssh.pl have unified interface, not depending on its backend.
# These options are mapping to specific options for each backend and
# it is configured by "conf/queue.conf" and "conf/slurm.conf" by default.
# If jobs failed, your configuration might be wrong for your environment.
#
#
# The official documentation for run.pl, queue.pl, slurm.pl, and ssh.pl:
# "Parallelization in Kaldi": http://kaldi-asr.org/doc/queue.html
# =========================================================~


# Select the backend used by run.sh from "local", "stdout", "sge", "slurm", or "ssh"
cmd_backend='local'

# Local machine, without any Job scheduling system
if [ "${cmd_backend}" = local ]; then

# The other usage
export train_cmd="run.pl"
# Used for "*_train.py": "--gpu" is appended optionally by run.sh
export cuda_cmd="run.pl"
# Used for "*_recog.py"
export decode_cmd="run.pl"

# Local machine logging to stdout and log file, without any Job scheduling system
elif [ "${cmd_backend}" = stdout ]; then

# The other usage
export train_cmd="stdout.pl"
# Used for "*_train.py": "--gpu" is appended optionally by run.sh
export cuda_cmd="stdout.pl"
# Used for "*_recog.py"
export decode_cmd="stdout.pl"


# "qsub" (Sun Grid Engine, or derivation of it)
elif [ "${cmd_backend}" = sge ]; then
# The default setting is written in conf/queue.conf.
# You must change "-q g.q" for the "queue" for your environment.
# To know the "queue" names, type "qhost -q"
# Note that to use "--gpu *", you have to setup "complex_value" for the system scheduler.

export train_cmd="queue.pl"
export cuda_cmd="queue.pl"
export decode_cmd="queue.pl"


# "qsub" (Torque/PBS.)
elif [ "${cmd_backend}" = pbs ]; then
# The default setting is written in conf/pbs.conf.

export train_cmd="pbs.pl"
export cuda_cmd="pbs.pl"
export decode_cmd="pbs.pl"


# "sbatch" (Slurm)
elif [ "${cmd_backend}" = slurm ]; then
# The default setting is written in conf/slurm.conf.
# You must change "-p cpu" and "-p gpu" for the "partition" for your environment.
# To know the "partion" names, type "sinfo".
# You can use "--gpu * " by default for slurm and it is interpreted as "--gres gpu:*"
# The devices are allocated exclusively using "${CUDA_VISIBLE_DEVICES}".

export train_cmd="slurm.pl"
export cuda_cmd="slurm.pl"
export decode_cmd="slurm.pl"

elif [ "${cmd_backend}" = ssh ]; then
# You have to create ".queue/machines" to specify the host to execute jobs.
# e.g. .queue/machines
# host1
# host2
# host3
# Assuming you can login them without any password, i.e. You have to set ssh keys.

export train_cmd="ssh.pl"
export cuda_cmd="ssh.pl"
export decode_cmd="ssh.pl"

# This is an example of specifying several unique options in the JHU CLSP cluster setup.
# Users can modify/add their own command options according to their cluster environments.
elif [ "${cmd_backend}" = jhu ]; then

export train_cmd="queue.pl --mem 2G"
export cuda_cmd="queue-freegpu.pl --mem 2G --gpu 1 --config conf/queue.conf"
export decode_cmd="queue.pl --mem 4G"

else
echo "$0: Error: Unknown cmd_backend=${cmd_backend}" 1>&2
return 1
fi
2 changes: 2 additions & 0 deletions egs2/iwslt24_indic/st1/conf/fbank.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
--sample-frequency=16000
--num-mel-bins=80
11 changes: 11 additions & 0 deletions egs2/iwslt24_indic/st1/conf/pbs.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Default configuration
command qsub -V -v PATH -S /bin/bash
option name=* -N $0
option mem=* -l mem=$0
option mem=0 # Do not add anything to qsub_opts
option num_threads=* -l ncpus=$0
option num_threads=1 # Do not add anything to qsub_opts
option num_nodes=* -l nodes=$0:ppn=1
default gpu=0
option gpu=0
option gpu=* -l ngpus=$0
1 change: 1 addition & 0 deletions egs2/iwslt24_indic/st1/conf/pitch.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
--sample-frequency=16000
12 changes: 12 additions & 0 deletions egs2/iwslt24_indic/st1/conf/queue.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Default configuration
command qsub -v PATH -cwd -S /bin/bash -j y -l arch=*64*
option name=* -N $0
option mem=* -l mem_free=$0,ram_free=$0
option mem=0 # Do not add anything to qsub_opts
option num_threads=* -pe smp $0
option num_threads=1 # Do not add anything to qsub_opts
option max_jobs_run=* -tc $0
option num_nodes=* -pe mpi $0 # You must set this PE as allocation_rule=1
default gpu=0
option gpu=0
option gpu=* -l gpu=$0 -q g.q
14 changes: 14 additions & 0 deletions egs2/iwslt24_indic/st1/conf/slurm.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Default configuration
command sbatch --export=PATH
option name=* --job-name $0
option time=* --time $0
option mem=* --mem-per-cpu $0
option mem=0
option num_threads=* --cpus-per-task $0
option num_threads=1 --cpus-per-task 1
option num_nodes=* --nodes $0
default gpu=0
option gpu=0 -p cpu
option gpu=* -p gpu --gres=gpu:$0 -c $0 # Recommend allocating more CPU than, or equal to the number of GPU
# note: the --max-jobs-run option is supported as a special case
# by slurm.pl and you don't have to handle it in the config file.
6 changes: 6 additions & 0 deletions egs2/iwslt24_indic/st1/conf/tuning/decode_st_conformer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
batch_size: 1
beam_size: 10
penalty: 0.0
maxlenratio: 0.0
minlenratio: 0.0
lm_weight: 0.0
89 changes: 89 additions & 0 deletions egs2/iwslt24_indic/st1/conf/tuning/train_st_conformer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
batch_type: numel
batch_bins: 25000000
accum_grad: 2
max_epoch: 80
patience: none
init: none
best_model_criterion:
- - valid
- acc
- max
keep_nbest_models: 10

encoder: conformer
encoder_conf:
output_size: 256
attention_heads: 4
linear_units: 2048
num_blocks: 12
dropout_rate: 0.1
positional_dropout_rate: 0.1
attention_dropout_rate: 0.1
input_layer: conv2d
normalize_before: true
macaron_style: true
rel_pos_type: latest
pos_enc_layer_type: rel_pos
selfattention_layer_type: rel_selfattn
activation_type: swish
use_cnn_module: true
cnn_module_kernel: 31

decoder: transformer
decoder_conf:
attention_heads: 4
linear_units: 2048
num_blocks: 6
dropout_rate: 0.1
positional_dropout_rate: 0.1
self_attention_dropout_rate: 0.1
src_attention_dropout_rate: 0.1

extra_asr_decoder: transformer
extra_asr_decoder_conf:
input_layer: embed
num_blocks: 6
linear_units: 2048
dropout_rate: 0.1

extra_mt_decoder: transformer
extra_mt_decoder_conf:
input_layer: embed
num_blocks: 2
linear_units: 2048
dropout_rate: 0.1

model_conf:
asr_weight: 0.3
mt_weight: 0.0
mtlalpha: 0.3
lsm_weight: 0.1
length_normalized_loss: false

optim: adam
optim_conf:
lr: 0.002
weight_decay: 0.000001
scheduler: warmuplr
scheduler_conf:
warmup_steps: 25000

frontend_conf:
n_fft: 400
hop_length: 160

specaug: specaug
specaug_conf:
apply_time_warp: true
time_warp_window: 5
time_warp_mode: bicubic
apply_freq_mask: true
freq_mask_width_range:
- 0
- 27
num_freq_mask: 2
apply_time_mask: true
time_mask_width_ratio_range:
- 0.
- 0.05
num_time_mask: 5
1 change: 1 addition & 0 deletions egs2/iwslt24_indic/st1/db.sh
Loading

0 comments on commit 435803c

Please sign in to comment.