Etymo Trending

Presented below is a list of the most trending papers added in the last two weeks.

• Topology of Learning in Artificial Neural Networks:
  The authors study the emergence of structure in the weights by applying methods from topological data analysis. They train simple feedforward neural networks on the MNIST dataset and monitor the evolution of the weights. When initialized to zero, the weights follow trajectories that branch off recurrently, thus generating trees that describe the growth of the effective capacity of each layer. When initialized to tiny random values, the weights evolve smoothly along two-dimensional surfaces. They show that natural coordinates on these learning surfaces correspond to important factors of variation.


• An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models:
  This paper presents a conceptually simple and effective transfer learning approach that combines the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. The method does not require pretraining or finetuning separate components of the network and models can be trained end-to-end in a single step.


• Seven Myths in Machine Learning Research:
  In this research paper, the authors present seven myths commonly believed to be true in machine learning research:
  1. Myth 1: TensorFlow is a Tensor manipulation library
  2. Myth 2: Image datasets are representative of real images found in the wild
  3. Myth 3: Machine Learning researchers do not use the test set for validation
  4. Myth 4: Every datapoint is used in training a neural network
  5. Myth 5: We need (batch) normalization to train very deep residual networks
  6. Myth 6: Attention > Convolution
  7. Myth 7: Saliency maps are robust ways to interpret neural networks