22nd February - 7th March 2019

1941 new papers

In this newsletter from Etymo, you can find out the latest development in machine learning research, including the most popular datasets used, the most frequently appearing keywords, the important research papers associated with the keywords, and the most trending papers in the past two weeks.

If you and your friends like this newsletter, you can subscribe to our fortnightly newsletters here.

Fortnight Summary

There are 1941 papers published in the past two weeks. Computer vision (CV) is still a main research area, as reflected on the popularity of the CV datasets and the most trending papers.

We present the emerging interests in research under the "Trending Phrases" section. The papers in this section show some cutting edge results. There are four good papers, each of which is related to Process Mining, Time Parameter, and Causal Graph.

Other notable development in research includes the following:

  • The topology of weight evolution in neural networks: Topology of Learning in Artificial Neural Networks
  • A conceptually simple and effective transfer learning approach without pretraining or finetuning: An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models
  • A study on the quantum learnability of constant-depth classical circuits under the uniform distribution and in the distribution-independent framework of probably approximately correct learning: Quantum hardness of learning shallow classical circuits
  • A standard pruning technique to uncover subnetworks whose initializations made them capable of effective training: The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
  • To tackle the model update problem with the recurrent meta-learning framework: Learning to Update for Object Tracking with Recurrent Meta-learner
  • A new threat model by characterizing, developing and evaluating new attacks in the brokered learning setting, along with new defenses for these attacks: Dancing in the Dark: Private Multi-Party Machine Learning in an Untrusted Setting
  • Probabilistic Modeling for Novelty Detection with Applications to Fraud Identification
  • Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
  • A deep learning algorithm with Monte Carlo (MC) and Quasi-Monte Carlo (QMC) methods to efficiently compute uncertainty propagation for nonlinear PDEs: Deep learning observables in computational fluid dynamics
  • An investigation on the model inversion problem in the adversarial settings: Adversarial Neural Network Inversion via Auxiliary Knowledge Alignment

  • There is also one interesting paper discussing the most common myths in machine learning:
  • Seven Myths in Machine Learning Research
  • Popular Datasets

    Computer vision is still the main focus area of research.

    Name Type Number of Papers
    MNIST Handwritten Digits 72
    ImageNet Image Dataset 54
    CIFAR-10 Tiny Image Dataset in 10 Classes 41
    COCO Common Objects in Context 27
    CelebA Large-scale CelebFaces Attributes 19
    KITTI Autonomous Driving 14
    Cityscapes Images from 50 different cities 11

    Trending Phrases

    In this section, we present a list of phrases that appeared significantly more in this newsletter than the previous newsletters.

    Etymo Trending

    Presented below is a list of the most trending papers added in the last two weeks.

    • Topology of Learning in Artificial Neural Networks:
      The authors study the emergence of structure in the weights by applying methods from topological data analysis. They train simple feedforward neural networks on the MNIST dataset and monitor the evolution of the weights. When initialized to zero, the weights follow trajectories that branch off recurrently, thus generating trees that describe the growth of the effective capacity of each layer. When initialized to tiny random values, the weights evolve smoothly along two-dimensional surfaces. They show that natural coordinates on these learning surfaces correspond to important factors of variation.

    • An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models:
      This paper presents a conceptually simple and effective transfer learning approach that combines the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. The method does not require pretraining or finetuning separate components of the network and models can be trained end-to-end in a single step.

    • Seven Myths in Machine Learning Research:
      In this research paper, the authors present seven myths commonly believed to be true in machine learning research:
      1. Myth 1: TensorFlow is a Tensor manipulation library
      2. Myth 2: Image datasets are representative of real images found in the wild
      3. Myth 3: Machine Learning researchers do not use the test set for validation
      4. Myth 4: Every datapoint is used in training a neural network
      5. Myth 5: We need (batch) normalization to train very deep residual networks
      6. Myth 6: Attention > Convolution
      7. Myth 7: Saliency maps are robust ways to interpret neural networks

    Frequent Words

    "Learning", "Model", "Data" and "Training" are the most frequent words. The top two papers associated with each of the key words are:

    Hope you have enjoyed this newsletter! If you have any comments or suggestions, please email or