Skip to content

Latest commit

 

History

History
1030 lines (794 loc) · 88.6 KB

BERT.md

File metadata and controls

1030 lines (794 loc) · 88.6 KB
Raw

Transfer-related Papers

a list of BERT-related papers. Any feedback is welcome.

Table of Contents

Downstream task

QA, MC, Dialogue

Slot filling

Analysis

Word segmentation, parsing, NER

Pronoun/coreference resolution

Sentiment analysis

Relation extraction

Knowledge base

Text classification

WSC, WNLI, NLI

Commonsense

Extractive summarization

IR

Generation

Modification (multi-task, masking strategy, etc.)

Transformer variants

Probe

Inside BERT

Multi-lingual

Other than English models

Domain specific

Multi-modal

Model compression

Misc.

collect BERT related resources.

Papers:

  1. arXiv:1810.04805, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

  2. arXiv:1812.06705, Conditional BERT Contextual Augmentation, Authors: Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, Songlin Hu

  3. arXiv:1812.03593, SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering, Authors: Chenguang Zhu, Michael Zeng, Xuedong Huang

  4. arXiv:1901.02860, Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, Authors: Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le and Ruslan Salakhutdinov.

  5. arXiv:1901.04085, Passage Re-ranking with BERT, Authors: Rodrigo Nogueira, Kyunghyun Cho

  6. arXiv:1902.02671, BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning, Authors: Asa Cooper Stickland, Iain Murray

  7. arXiv:1904.02232, BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis, Authors: Hu Xu, Bing Liu, Lei Shu, Philip S. Yu, [code]

Github Repositories:

official implement:

  1. google-research/bert, officical TensorFlow code and pre-trained models for BERT ,

implement of BERT besides tensorflow:

  1. codertimo/BERT-pytorch, Google AI 2018 BERT pytorch implementation,

  2. huggingface/pytorch-pretrained-BERT, A PyTorch implementation of Google AI's BERT model with script to load Google's pre-trained models,

  3. Separius/BERT-keras, Keras implementation of BERT with pre-trained weights,

  4. soskek/bert-chainer, Chainer implementation of "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding",

  5. innodatalabs/tbert, PyTorch port of BERT ML model

  6. guotong1988/BERT-tensorflow, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  7. dreamgonfly/BERT-pytorch, PyTorch implementation of BERT in "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

  8. CyberZHG/keras-bert, Implementation of BERT that could load official pre-trained models for feature extraction and prediction

  9. soskek/bert-chainer, Chainer implementation of "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

  10. MaZhiyuanBUAA/bert-tf1.4.0, bert-tf1.4.0

  11. dhlee347/pytorchic-bert, Pytorch Implementation of Google BERT,

  12. kpot/keras-transformer, Keras library for building (Universal) Transformers, facilitating BERT and GPT models,

  13. miroozyx/BERT_with_keras, A Keras version of Google's BERT model,

  14. conda-forge/pytorch-pretrained-bert-feedstock, A conda-smithy repository for pytorch-pretrained-bert. ,

  15. Rshcaroline/BERT_Pytorch_fastNLP, A PyTorch & fastNLP implementation of Google AI's BERT model.

  16. nghuyong/ERNIE-Pytorch, ERNIE Pytorch Version,

  17. dmlc/gluon-nlp, Gluon + MXNet implementation that reproduces BERT pretraining and finetuning on GLUE benchmark, SQuAD, etc,

  18. dbiir/UER-py, UER-py is a toolkit for pre-training on general-domain corpus and fine-tuning on downstream task. UER-py maintains model modularity and supports research extensibility. It facilitates the use of different pre-training models (e.g. BERT), and provides interfaces for users to further extend upon.

improvement over BERT:

  1. thunlp/ERNIE, Source code and dataset for ACL 2019 paper "ERNIE: Enhanced Language Representation with Informative Entities", imporove bert with heterogeneous information fusion.

  2. PaddlePaddle/LARK, LAnguage Representations Kit, PaddlePaddle implementation of BERT. It also contains an improved version of BERT, ERNIE, for chinese NLP tasks.

  3. ymcui/Chinese-BERT-wwm, Pre-Training with Whole Word Masking for Chinese BERT https://arxiv.org/abs/1906.08101,

  4. zihangdai/xlnet, XLNet: Generalized Autoregressive Pretraining for Language Understanding,

  5. kimiyoung/transformer-xl, Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, This repository contains the code in both PyTorch and TensorFlow for our paper.

  6. GaoPeng97/transformer-xl-chinese, （transformer xl for text generation of chinese）,

other resources for BERT:

  1. brightmart/bert_language_understanding, Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN,

  2. Y1ran/NLP-BERT--ChineseVersion,

  3. yangbisheng2009/cn-bert,

  4. JayYip/bert-multiple-gpu, A multiple GPU support version of BERT,

  5. HighCWu/keras-bert-tpu, Implementation of BERT that could load official pre-trained models for feature extraction and prediction on TPU,

  6. Willyoung2017/Bert_Attempt, PyTorch Pretrained Bert,

  7. Pydataman/bert_examples, some examples of bert, run_classifier.py

  8. guotong1988/BERT-chinese, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  9. zhongyunuestc/bert_multitask, 多任务task

  10. Microsoft/AzureML-BERT, End-to-end walk through for fine-tuning BERT using Azure Machine Learning ,

  11. bigboNed3/bert_serving, export bert model for serving,

  12. yoheikikuta/bert-japanese, BERT with SentencePiece for Japanese text.

  13. whqwill/seq2seq-keyphrase-bert, add BERT to encoder part for https://github.com/memray/seq2seq-keyphrase-pytorch,

  14. algteam/bert-examples, bert-demo,

  15. cedrickchee/awesome-bert-nlp, A curated list of NLP resources focused on BERT, attention mechanism, Transformer networks, and transfer learning.

  16. cnfive/cnbert,

  17. brightmart/bert_customized, bert with customized features,

  18. JayYip/bert-multitask-learning, BERT for Multitask Learning,

  19. yuanxiaosc/BERT_Paper_Chinese_Translation, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 。Chinese Translation! https://yuanxiaosc.github.io/2018/12/…,

  20. yaserkl/BERTvsULMFIT, Comparing Text Classification results using BERT embedding and ULMFIT embedding,

  21. kpot/keras-transformer, Keras library for building (Universal) Transformers, facilitating BERT and GPT models,

  22. 1234560o/Bert-model-code-interpretation,

  23. cdathuraliya/bert-inference, A helper class for Google BERT (Devlin et al., 2018) to support online prediction and model pipelining.

  24. gameofdimension/java-bert-predict, turn bert pretrain checkpoint into saved model for a feature extracting demo in java

  25. 1234560o/Bert-model-code-interpretation,

domain specific BERT:

  1. allenai/scibert, A BERT model for scientific text. https://arxiv.org/abs/1903.10676,

  2. MeRajat/SolvingAlmostAnythingWithBert, BioBert Pytorch

  3. kexinhuang12345/clinicalBERT, ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission https://arxiv.org/abs/1904.05342

  4. EmilyAlsentzer/clinicalBERT, repository for Publicly Available Clinical BERT Embeddings

BERT Deploy Tricks:

  1. zhihu/cuBERT, Fast implementation of BERT inference directly on NVIDIA (CUDA, CUBLAS) and Intel MKL

  2. xmxoxo/BERT-train2deploy, Bert Model training and deploy,

BERT QA & RC task:

  1. sogou/SMRCToolkit, This toolkit was designed for the fast and efficient development of modern machine comprehension models, including both published models and original prototypes.,

  2. benywon/ChineseBert, This is a chinese Bert model specific for question answering,

  3. matthew-z/R-net, R-net in PyTorch, with BERT and ELMo,

  4. nyu-dl/dl4marco-bert, Passage Re-ranking with BERT,

  5. xzp27/BERT-for-Chinese-Question-Answering,

  6. chiayewken/bert-qa, BERT for question answering starting with HotpotQA,

  7. ankit-ai/BertQA-Attention-on-Steroids, BertQA - Attention on Steroids,

  8. NoviScl/BERT-RACE, This work is based on Pytorch implementation of BERT (https://github.com/huggingface/pytorch-pretrained-BERT). I adapted the original BERT model to work on multiple choice machine comprehension.

  9. eva-n27/BERT-for-Chinese-Question-Answering,

  10. allenai/allennlp-bert-qa-wrapper, This is a simple wrapper on top of pretrained BERT based QA models from pytorch-pretrained-bert to make AllenNLP model archives, so that you can serve demos from AllenNLP.

  11. edmondchensj/ChineseQA-with-BERT, EECS 496: Advanced Topics in Deep Learning Final Project: Chinese Question Answering with BERT (Baidu DuReader Dataset)

  12. graykode/toeicbert, TOEIC(Test of English for International Communication) solving using pytorch-pretrained-BERT model.,

  13. graykode/KorQuAD-beginner, https://github.com/graykode/KorQuAD-beginner

  14. krishna-sharma19/SBU-QA, This repository uses pretrain BERT embeddings for transfer learning in QA domain

BERT classification task:

  1. zhpmatrix/Kaggle-Quora-Insincere-Questions-Classification,

  2. maksna/bert-fine-tuning-for-chinese-multiclass-classification, use google pre-training model bert to fine-tuning for the chinese multiclass classification

  3. NLPScott/bert-Chinese-classification-task,

  4. Socialbird-AILab/BERT-Classification-Tutorial,

  5. fooSynaptic/BERT_classifer_trial, BERT trial for chinese corpus classfication

  6. xiaopingzhong/bert-finetune-for-classfier,

  7. pengming617/bert_classification, ,

  8. xieyufei1993/Bert-Pytorch-Chinese-TextClassification, Pytorch Bert Finetune in Chinese Text Classification,

  9. liyibo/text-classification-demos, Neural models for Text Classification in Tensorflow, such as cnn, dpcnn, fasttext, bert ...,

  10. circlePi/BERT_Chinese_Text_Class_By_pytorch, A Pytorch implements of Chinese text class based on BERT_Pretrained_Model,

  11. kaushaltrivedi/bert-toxic-comments-multilabel, Multilabel classification for Toxic comments challenge using Bert,

  12. lonePatient/BERT-chinese-text-classification-pytorch, This repo contains a PyTorch implementation of a pretrained BERT model for text classification.,

BERT Sentiment Analysis

  1. Chung-I/Douban-Sentiment-Analysis, Sentiment Analysis on Douban Movie Short Comments Dataset using BERT.

  2. lynnna-xu/bert_sa, bert sentiment analysis tensorflow serving with RESTful API

  3. HSLCY/ABSA-BERT-pair, Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence (NAACL 2019) https://arxiv.org/abs/1903.09588,

  4. songyouwei/ABSA-PyTorch, Aspect Based Sentiment Analysis, PyTorch Implementations.,

  5. howardhsu/BERT-for-RRC-ABSA, code for our NAACL 2019 paper: "BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis",

  6. brightmart/sentiment_analysis_fine_grain, Multi-label Classification with BERT; Fine Grained Sentiment Analysis from AI challenger,

BERT NER task:

  1. zhpmatrix/bert-sequence-tagging,

  2. kyzhouhzau/BERT-NER, Use google BERT to do CoNLL-2003 NER ! ,

  3. king-menin/ner-bert, NER task solution (bert-Bi-LSTM-CRF) with google bert https://github.com/google-research.

  4. macanv/BERT-BiLSMT-CRF-NER, Tensorflow solution of NER task Using BiLSTM-CRF model with Google BERT Fine-tuning ,

  5. FuYanzhe2/Name-Entity-Recognition, Lstm-crf,Lattice-CRF,bert-ner

  6. mhcao916/NER_Based_on_BERT, this project is based on google bert model, which is a Chinese NER

  7. ProHiryu/bert-chinese-ner,

  8. sberbank-ai/ner-bert, BERT-NER (nert-bert) with google bert,

  9. kyzhouhzau/Bert-BiLSTM-CRF, This model base on bert-as-service. Model structure : bert-embedding bilstm crf. ,

  10. Hoiy/berserker, Berserker - BERt chineSE woRd toKenizER, Berserker (BERt chineSE woRd toKenizER) is a Chinese tokenizer built on top of Google's BERT model. ,

  11. Kyubyong/bert_ner, Ner with Bert,

  12. jiangpinglei/BERT_ChineseWordSegment, A Chinese word segment model based on BERT, F1-Score 97%,

  13. yanwii/ChineseNER,

  14. lemonhu/NER-BERT-pytorch, PyTorch solution of NER task Using Google AI's pre-trained BERT model.

BERT Text Summarization Task:

  1. nlpyang/BertSum, Code for paper Fine-tune BERT for Extractive Summarization,

  2. santhoshkolloju/Abstractive-Summarization-With-Transfer-Learning, Abstractive summarisation using Bert as encoder and Transformer Decoder,

  3. nayeon7lee/bert-summarization, Implementation of 'Pretraining-Based Natural Language Generation for Text Summarization', Paper: https://arxiv.org/pdf/1902.09243.pdf

  4. dmmiller612/lecture-summarizer, Lecture summarizer with BERT

BERT Text Generation Task:

  1. asyml/texar, Toolkit for Text Generation and Beyond https://texar.io, Texar is a general-purpose text generation toolkit, has also implemented BERT here for classification, and text generation applications by combining with Texar's other modules.

  2. voidful/BertGenerate, Fine tuning bert for text generation,

  3. Tiiiger/bert_score, BERT score for language generation,

BERT Knowledge Graph Task :

  1. lvjianxin/Knowledge-extraction,

  2. sakuranew/BERT-AttributeExtraction, USING BERT FOR Attribute Extraction in KnowledgeGraph. fine-tuning and feature extraction.,

  3. aditya-AI/Information-Retrieval-System-using-BERT,

  4. jkszw2014/bert-kbqa-NLPCC2017, A trial of kbqa based on bert for NLPCC2016/2017 Task 5, https://blog.csdn.net/ai_1046067944/article/details/86707784 ,

  5. yuanxiaosc/Schema-based-Knowledge-Extraction, Code for http://lic2019.ccf.org.cn/kg,

  6. yuanxiaosc/Entity-Relation-Extraction, Entity and Relation Extraction Based on TensorFlow.Schema based Knowledge Extraction, SKE 2019 http://lic2019.ccf.org.cn,

  7. WenRichard/KBQA-BERT, https://zhuanlan.zhihu.com/p/62946533 ,

BERT Coreference Resolution

  1. ianycxu/RGCN-with-BERT, Gated-Relational Graph Convolutional Networks (RGCN) with BERT for Coreference Resolution Task

  2. isabellebouchard/BERT_for_GAP-coreference, BERT finetuning for GAP unbiased pronoun resolution

BERT visualization toolkit:

  1. jessevig/bertviz, Tool for visualizing BERT's attention,

BERT chatbot :

  1. GaoQ1/rasa_nlu_gq, turn natural language into structured data,

  2. GaoQ1/rasa_chatbot_cn,

  3. GaoQ1/rasa-bert-finetune,

  4. geodge831012/bert_robot

  5. yuanxiaosc/BERT-for-Sequence-Labeling-and-Text-Classification, This is the template code to use BERT for sequence lableing and text classification, in order to facilitate BERT for more tasks. Currently, the template code has included conll-2003 named entity identification, Snips Slot Filling and Intent Prediction.

  6. guillaume-chevalier/ReuBERT, A question-answering chatbot, simply.

BERT language model and embedding:

  1. hanxiao/bert-as-service, Mapping a variable-length sentence to a fixed-length vector using pretrained BERT model,

  2. YC-wind/embedding_study,

  3. Kyubyong/bert-token-embeddings, Bert Pretrained Token Embeddings,

  4. xu-song/bert_as_language_model, bert as language model, fork from https://github.com/google-research/bert,

  5. yuanxiaosc/Deep_dynamic_word_representation, TensorFlow code and pre-trained models for deep dynamic word representation (DDWR). It combines the BERT model and ELMo's deep context word representation.,

  6. imgarylai/bert-embedding, Token level embeddings from BERT model on mxnet and gluonnlp http://bert-embedding.readthedocs.io/,

  7. terrifyzhao/bert-utils,

  8. fennuDetudou/BERT_implement,

  9. whqwill/seq2seq-keyphrase-bert, add BERT to encoder part for https://github.com/memray/seq2seq-keyphrase-pytorch,

  10. charles9n/bert-sklearn, a sklearn wrapper for Google's BERT model,

  11. NVIDIA/Megatron-LM, Ongoing research training transformer language models at scale, including: BERT,

  12. hankcs/BERT-token-level-embedding, Generate BERT token level embedding without pain

BERT Text Match:

  1. pengming617/bert_textMatching, 利用预训练的中文模型实现基于bert的语义匹配模型 数据集为LCQMC官方数据

  2. Brokenwind/BertSimilarity, Computing similarity of two sentences with google's BERT algorithm

  3. policeme/chinese_bert_similarity, bert chinese similarity

  4. lonePatient/bert-sentence-similarity-pytorch, This repo contains a PyTorch implementation of a pretrained BERT model for sentence similarity task.

  5. nouhadziri/DialogEntailment, The implementation of the paper "Evaluating Coherence in Dialogue Systems using Entailment" https://arxiv.org/abs/1904.03371

ko bert

https://github.com/jeongukjae/KR-BERT-SimCSE

BERT tutorials:

  1. graykode/nlp-tutorial, Natural Language Processing Tutorial for Deep Learning Researchers https://www.reddit.com/r/MachineLearn…,

  2. dragen1860/TensorFlow-2.x-Tutorials, TensorFlow 2.x version's Tutorials and Examples, including CNN, RNN, GAN, Auto-Encoders, FasterRCNN, GPT, BERT examples, etc. TF 2.0。,

한국어 sentence bert 모델

논문 요약: https://www.marktechpost.com/.../researchers-from-china.../

종이: https://arxiv.org/pdf/2207.07116v1.pdf Github: https://github.com/lightdxy/bootmae

Time Series Related Survey

  • Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects, in arXiv 2023. [paper] [Website]
  • A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection, in arXiv 2023. [paper] [Website]
  • Time series data augmentation for deep learning: a survey, in IJCAI 2021. [paper]
  • Neural temporal point processes: a review, in IJCAI 2021. [paper]
  • Time-series forecasting with deep learning: a survey, in Philosophical Transactions of the Royal Society A 2021. [paper]
  • Deep learning for time series forecasting: a survey, in Big Data 2021. [paper]
  • Neural forecasting: Introduction and literature overview, in arXiv 2020. [paper]
  • Deep learning for anomaly detection in time-series data: review, analysis, and guidelines, in Access 2021. [paper]
  • A review on outlier/anomaly detection in time series data, in ACM Computing Surveys 2021. [paper]
  • A unifying review of deep and shallow anomaly detection, in Proceedings of the IEEE 2021. [paper]
  • Deep learning for time series classification: a review, in Data Mining and Knowledge Discovery 2019. [paper]
  • More related time series surveys, tutorials, and papers can be found at this repo.

Application Domains of Time Series Transformers

Transformers in Forecasting

Time Series Forecasting

  • Make Transformer Great Again for Time Series Forecasting: Channel Aligned Robust Dual Transformer, in arXiv 2023. [paper]
  • A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, in ICLR 2023. [paper] [code]
  • Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting, in ICLR 2023. [paper]
  • Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting, in ICLR 2023. [paper]
  • Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting, in NeurIPS 2022. [paper]
  • Learning to Rotate: Quaternion Transformer for Complicated Periodical Time Series Forecasting”, in KDD 2022. [paper]
  • FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, in ICML 2022. [paper] [official code]
  • TACTiS: Transformer-Attentional Copulas for Time Series, in ICML 2022. [paper]
  • Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting, in ICLR 2022. [paper] [official code]
  • Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, in NeurIPS 2021. [paper] [official code]
  • Informer: Beyond efficient transformer for long sequence time-series forecasting, in AAAI 2021. [paper] [official code] [dataset]
  • Temporal fusion transformers for interpretable multi-horizon time series forecasting, in International Journal of Forecasting 2021. [paper] [code]
  • Probabilistic Transformer For Time Series Analysis, in NeurIPS 2021. [paper]
  • Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case, in arXiv 2020. [paper]
  • Adversarial sparse transformer for time series forecasting, in NeurIPS 2020. [paper] [code]
  • Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, in NeurIPS 2019. [paper] [code]
  • SSDNet: State Space Decomposition Neural Network for Time Series Forecasting, in ICDM 2021, [paper]
  • From Known to Unknown: Knowledge-guided Transformer for Time-Series Sales Forecasting in Alibaba, in arXiv 2021. [paper]
  • TCCT: Tightly-coupled convolutional transformer on time series forecasting, in Neurocomputing 2022. [paper]
  • Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting, in IJCAI 2022. [paper]

Spatio-Temporal Forecasting

  • AirFormer: Predicting Nationwide Air Quality in China with Transformers, in AAAI 2023. [paper] [official code]
  • Earthformer: Exploring Space-Time Transformers for Earth System Forecasting, in NeurIPS 2022. [paper] [official code]
  • Bidirectional Spatial-Temporal Adaptive Transformer for Urban Traffic Flow Forecasting, in TNNLS 2022. [paper]
  • Spatio-temporal graph transformer networks for pedestrian trajectory prediction, in ECCV 2020. [paper] [official code]
  • Spatial-temporal transformer networks for traffic flow forecasting, in arXiv 2020. [paper] [official code]
  • Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting, in Transactions in GIS 2022. [paper]

Event Forecasting

Transformers in Anomaly Detection

  • CAT: Beyond Efficient Transformer for Content-Aware Anomaly Detection in Event Sequences, in KDD 2022. [paper] [official code]
  • DCT-GAN: Dilated Convolutional Transformer-based GAN for Time Series Anomaly Detection, in TKDE 2022. [paper]
  • Concept Drift Adaptation for Time Series Anomaly Detection via Transformer, in Neural Processing Letters 2022. [paper]
  • Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy, in ICLR 2022. [paper] [official code]
  • TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data, in VLDB 2022. [paper] [official code]
  • Learning graph structures with transformer for multivariate time series anomaly detection in IoT, in IEEE Internet of Things Journal 2021. [paper] [official code]
  • Spacecraft Anomaly Detection via Transformer Reconstruction Error, in ICASSE 2019. [paper]
  • Unsupervised Anomaly Detection in Multivariate Time Series through Transformer-based Variational Autoencoder, in CCDC 2021. [paper]
  • Variational Transformer-based anomaly detection approach for multivariate time series, in Measurement 2022. [paper]

Transformers in Classification

  • TrajFormer: Efficient Trajectory Classification with Transformers, in CIKM 2022. [paper]
  • TARNet : Task-Aware Reconstruction for Time-Series Transformer, in KDD 2022. [paper] [official code]
  • A transformer-based framework for multivariate time series representation learning, in KDD 2021. [paper] [official code]
  • Voice2series: Reprogramming acoustic models for time series classification, in ICML 2021. [paper] [official code]
  • Gated Transformer Networks for Multivariate Time Series Classification, in arXiv 2021. [paper] [official code]
  • Self-attention for raw optical satellite time series classification, in ISPRS Journal of Photogrammetry and Remote Sensing 2020. [paper] [official code]
  • Self-supervised pretraining of transformers for satellite image time series classification, in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2020. [paper]
  • Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series, in ACM TKDD 2022. [paper] [official code]

canvers-ko2en

840만 번역쌍으로 bart 기반으로 튜닝이 되어 있습니다. GPU가 있다면, transformers 에서 flash attention 2 와 사용하실 수도 있고 ctranslate2 버전도 있어 cpu에서도 충분히 빠르게 모델을 사용할 수도 있습니다. 저희 내부적인 전략이 영문 모델을 한글 튜닝하기 보다는 영문 모델을 기본으로 빠르게 follow-up 하되 앞뒤로 번역을 붙여서 쓰는 방식을 취하고 있는데, 관리차원에서 올린 모델중에 번역모델들이 꾸준하게 다운로드가 되고 있긴 하더라구요. https://huggingface.co/circulus/canvers-ko2en-v2 https://huggingface.co/circulus/canvers-en2ko-v2 https://huggingface.co/circulus/canvers-ko2en-ct2-v2 https://huggingface.co/circulus/canvers-en2ko-ct2-v2