ML_transfer.md

ML - Transfer Learning

Paper	Conference	Remarks
A Survey on Transfer Learning	TKDE 2009	1. This survey focuses on categorizing and reviewing the current progress on transfer learning for classification, regression and clustering problems. 2. Discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as co-variate shift.
A Kernel Two-Sample Test	JMLR 2012	1. Propose a framework for analyzing and comparing distributions, which we use to construct statistical tests to determine if two samples are drawn from different distributions. 2. The proposed test statistic is the largest difference in expectations over functions in the unit ball of a reproducing kernel Hilbert space (RKHS), and is called the maximum mean discrepancy (MMD).
Generative Moment Matching Networks	ICML 2015	1. Formulate a method that generates an independent sample via a single feedforward pass through a multilayer perceptron, as in the GAN. 2. Utilize a technique from statistical hypothesis testing known as maximum mean discrepancy (MMD), which leads to a simple objective that can be interpreted as matching all orders of statistics between a dataset and samples from the model, and can be trained by backpropagation. 3. Further boost the performance of this approach by combining our generative network with an auto-encoder network, using MMD to learn to generate codes that can then be decoded to produce samples.
Learning Transferable Features with Deep Adaptation Networks	ICML 2015	1. Propose a new Deep Adaptation Network (DAN) architecture, which generalizes deep convolutional neural network to the domain adaptation scenario, where hidden representations of all task-specific layers are embedded in a reproducing kernel Hilbert space where the mean embeddings of different domain distributions can be explicitly matched. 2. DAN can learn transferable features with statistical guarantees, and can scale linearly by unbiased estimate of kernel embedding.
Domain-Adversarial Training of Neural Networks	JMLR 2016	1. Introduce a new representation learning approach for domain adaptation, in which data at training and test time come from similar but different distributions. 2. The approach promotes the emergence of features that are (i) discriminative for the main learning task on the source domain and (ii) indiscriminate with respect to the shift between the domains. 3. Show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation and stochastic gradient descent, and can thus be implemented with little effort using any of the deep learning packages.
Adversarial Discriminative Domain Adaptation	CVPR 2017	1. Propose a general framework which combines discriminative modeling, untied weight sharing, and a GAN loss, called Adversarial Discriminative Domain Adaptation (ADDA). 2. Show that ADDA is more effective yet considerably simpler than competing domain-adversarial methods, and demonstrate the promise of our approach by exceeding state-of-the-art unsupervised adaptation results on standard cross-domain digit classification tasks and a new more difficult cross-modality object classification task.
AutoDIAL: Automatic DomaIn Alignment Layers	ICCV 2017	1. Propose to align the learned CNN representations by embedding in any given network specific Domain Alignment Layers, designed to match the source and target feature distributions to a reference one. 2. The method is able to automatically learn the degree of feature alignment required at different levels of the deep network.
Deep Transfer Learning with Joint Adaptation Networks	ICML 2017	1. Present joint adaptation networks (JAN), which learn a transfer network by aligning the joint distributions of multiple domain-specific layers across domains based on a joint maximum mean discrepancy (JMMD) criterion. 2. Adversarial training strategy is adopted to maximize JMMD such that the distributions of the source and target domains are made more distinguishable.
A DIRT-T Approach to Unsupervised Domain Adaptation	ICLR 2018	1. Based on the cluster assumption, i.e., decision boundaries should not cross high-density data regions. This paper proposes two novel and related models: 1) the Virtual Adversarial Domain Adaptation (VADA) model, which combines domain adversarial training with a penalty term that punishes the violation the cluster assumption; 2) the Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T) model, which takes the VADA model as initialization and employs natural gradient steps to further minimize the cluster assumption violation.
Co-regularized Alignment for Unsupervised Domain Adaptation	NIPS 2018	1. Propose co-regularized domain alignment for unsupervised domain adaptation, which constructs multiple diverse feature spaces and aligns source and target distributions in each of them individually, while encouraging that alignments agree with each other with regard to the class predictions on the unlabeled target examples. 2. The proposed method is generic and can be used to improve any domain adaptation method which uses domain alignment.
Wasserstein Distance Guided Representation Learning for Domain Adaptation	AAAI 2018	1. Propose a novel approach to learn domain invariant feature representations, namely Wasserstein Distance Guided Representation Learning (WDGRL), which utilizes a neural network, denoted by the domain critic, to estimate empirical Wasserstein distance between the source and target samples and optimizes the feature extractor network to minimize the estimated Wasserstein distance in an adversarial manner.
A Survey of Unsupervised Deep Domain Adaptation	Arxiv 2018	1. This survey compares unsupervised domain adaptation approaches by examining alternative methods, the unique and common elements, results, and theoretical insights.
Transfer Adaptation Learning: A Decade Survey	Arxiv 2019	1. Surveys the recent advances in transfer adaptation learning methodology and potential benchmarks. 2. Broader challenges being faced by transfer adaptation learning researchers are identified, i.e., instance re-weighting adaptation, feature adaptation, classifier adaptation, deep network adaptation, and adversarial adaptation, which are beyond the early semi-supervised and unsupervised split.