Icml 2019 When Can You See the Review
ICML 2019 Review
ICML is a huge conference, with over 6,000 researchers and students, and several concurrent tracks. The conference was followed past a series of workshops on Friday and Sabbatum, which were as well organized as the conference, if not amend.
Hither are some highlights related to our research at Numenta:
Continuous Learning
Natalia Dias-Rodriguez, representing ContinualAI, gave an interesting talk of the challenges in continuous learning. I'yard glad to hear they are working on a survey of continuous learning, to be released in about ii weeks in arXiv, excited to read it! (edit: now bachelor in arXiv). Marta White, from the University of Alberta, also presented some fascinating research on continuous learning and thin representations.
There were many other promising talks and posters on continuous learning, including a total-day workshop on lifelong learning and some other on multitask learning. It was not bad to run into the latest papers by the Berkeley grouping on unsupervised meta-learning and reinforcement learning without explicit advantage functions.
Robustness
Creating models robust to noise and adversarial attacks is a hot topic in ML, and ICML had several interesting papers. The talks covered both theoretical and empirical evaluation of adversarial attacks and defense methods, and dissonance and corruption robustness.
Ahmad and Scheinkman, from Numenta, showed how the combination of sparse connections and sparse activation part (k-winners) promotes high levels of sparsity and tin can lead to networks that are both ability-efficient and robust to noise. We are currently working on augmenting sparse neural networks with structural plasticity, and extending its application to larger and more complex models and datasets.
An interesting dataset, called ImageNet-P, was recently released by Hendrycks and Dietterich at ICLR. It features images augmented with 15 types of algorithmically generated corruptions, aimed at mimicking noisy data often found in real-world scenarios. An MNIST version of this dataset, chosen MNIST-C, was presented in ICML uncertainty and robustness workshop.
Thin representations
Our encephalon is incredibly sparse, and that is key to how we larn. Ahmad has already shown in a previous paper how sparse representations can pb to semantic representations better suited to build robust systems.
Pruning can likewise promote structural sparsity after grooming and lead to smaller networks, which can be embedded in smaller devices and are more often than not faster and more power-efficient. Two contempo papers on pruning published in ICLR, the Lottery Ticket Hypothesis (LT) and Rethinking the Value of Network Pruning, hints that some of the model's performance might be attributed to learning the structure of the network instead of learning the weights.
A more recent paper presented in ICML deep learning theory workshop, called Deconstructing the Lottery Ticket Hypothesis, takes a pace farther in that management. Zhou et al. extends the LT paper and shows that a pruned neural network, with weights initialized to constant values with the same sign as the weights of the original network, can accomplish up to 87% accuracy in MNIST with no training.
The conference and workshops also featured a handful of interesting talks and discussions on naturally emerging sparsity and implicit sparsity in neural networks trained with or without regularization.
Structural plasticity
A fundamental characteristic of our encephalon is how much information technology changes during the course of our lifetime or even in much smaller periods. In simply a few days, upwards to 30% of our synapses can exist replaced, which is a hallmark to how adjustable our encephalon is.
Zeroing out weights or activations is a common strategy during training (dropout and variants) and is a powerful regularization method. But can we build models that tin can also be sparse at inference time, while keeping the benefits of regularization? That is a compelling thought, being discussed in the community for years, with thought-provoking papers such equally Louizos et al.
A contempo paper that captured my involvement is the Thin Evolutionary Training (SET). In the SET model, the initial weights are sparsely distributed at around a 4% sparsity level. Connections are pruned during grooming, based on magnitude, and an equal number of random connections are reinitialized, promoting a construction search during training.
Ii follow-up papers on SET were presented at ICML. The idea of dynamic thin reparametrization is further explored in Mostafa and Wang, with an improved heuristic that reallocates more than connections to layers with higher grooming loss as opposed to random reallocation. The SET authors too nowadays a follow-upwards work that proposes pruning all incoming and outgoing connections of a neuron, showing improved results.
Others
Curriculum learning is a common technique in reinforcement learning and consists of presenting increasingly difficult tasks to your agent. This is especially useful in environments where the reward is thin and hard to obtain.
A paper by Mangalam and Prabhu, from UnifyID, compares which examples are learned faster by deep neural nets and shallow classifiers, and concludes that the goodness of an example is an aspect of the sample, and not of the model. This points to an interest direction of enquiry, to utilize shallow classifiers to distinguish easy to learn from hard to learn examples and apply it for curriculum learning in more complex models.
I also saw fascinating work on self-supervised learning models that uses contrastive predictive coding to pre-railroad train neural networks, allowing them to learn faster or with less labeled samples. It particularly caught my attention a paper past Löwe et al., from the University of Amsterdam, which proposes a novel deep learning method that does not require labels nor terminate-to-end backpropagation. It is certainly an interesting research management for biologically inspired learning models.
Wrapping up
I accept at to the lowest degree a dozen more highlights, which wouldn't fit the post! Overall I had a groovy fourth dimension at ICML, and hope to be there side by side year for another round of basis-breaking inquiry.
Source: https://medium.com/@lucasosouza/icml-2019-review-8379358c7805
Enviar um comentário for "Icml 2019 When Can You See the Review"