Paper Summary: “Rethinking Pre-training and Self-training”

Paper by: Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, Quoc V. Le


Pretraining is used all the time! We use weights pre-trained on ImageNet for all sorts of other models. But one paper showed that pre-training had a limited impact on COCO for object…

Paper Summary: “A Review of Algorithms and Hardware Implementations for Spiking Neural Networks”

Paper by: Duy-Anh Nguyen, Xuan-Tu Tran, Francesca Iacopi


Deep Learning (DL) has achieved a ton in recent years — from beating us at DOTA, to recognizing the speech of humans. Despite its awesome performance, DL costs a ton computationally — thus we cant put it into embedded applications.

One alternative…

Paper Summary: “Simple Model of Spiking Neurons”

The voltage of the Izhikevich Neuron Model. I created it myself and you can play around with it too!

Paper by: Eugene M. Izhikevich

Side Note — Paper Implementation:

I implemented the Izhikevich Neuron Model! You can check out my implementation of the network (with all the synaptic connections) here! I also implemented the neuron model itself with all the different neural dynamics! You can check it out here!


This paper presents a paper…

Paper Summary: “Evolving Spiking Neural Networks for Nonlinear Control Problems”

Paper by: Huanneng Qiu, Matthew Garratt, David Howard, Sreenatha Anavatti


Spiking Neural Networks (SNN) has attracted a lot of attention recently — but lots of the research done here are non-behavioural and discontinuous. But this paper presents an SNN recurrent net that can handle non-linear control problems in continuous domains.

Paper Summary: “A review of learning in biologically plausible spiking neural networks”

Paper by: Aboozar Taherkhania, Ammar Belatreche, Yuhua Li, Georgina Cosma, Liam P. Maguire, and T. M McGinnity


Artificial Neural Networks (ANNs) have shown their power in recent years in many areas like vision, robotics, and natural language processing. …

Paper Summary: “Deep Learning in Spiking Neural Networks”

Paper by: Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothee Masquelier, Anthony Maida


Artificial Neural Networks (ANNs) have dominated the ML world — they’re extremely powerful through the use of backpropagation, tons of data, and the simplicity of the neuron itself. But despite their name, ANN’s aren’t like our brains.

Paper Summary: “Deep Mutual Learning”

Paper by: Ying Zhang, Tao Xiang, Timothy M. Hospedales, Huchuan Lu


In conventional distillation, we have a powerful teacher model/ensemble, which transfers its knowledge to a student network — which has lower memory and/or is faster to meet requirements for deployment.

This paper takes a different approach to distillation. Instead…

Paper Summary: “Differentiable plasticity: training plastic neural networks with backpropagation”

Paper by: Thomas Miconi, Jeff Clune, Kenneth O. Stanley


In our brains, plasticity has allowed for efficient lifelong learning — so why don’t we adopt that for neural networks? But this paper takes a different approach and uses gradient descent for plasticity! They used it for RNN’s and were able…

Paper summary: “Deep generative design: integration of topology optimization and generative models”

Lots of wheels with different designs. All generated using GANs and TopOp


This paper proposes integrating Deep learning with the generative design! Models are able to create new wheels which are aesthetic but also optimized for engineering performance. The framework uses topology optimization + generative models in an iterative fashion.

This framework can generate lots of different designs by starting with a…

Paper Summary: “Evolution Strategies as a Scalable Alternative to Reinforcement Learning”


For a long time, Markov Decision Process (MDP) based RL techniques have dominated. But this paper explores an alternative → Evolution Strategies (ES). They experiment on MuJoCo and Atari — and ES is competitive with RL!

ES scales very well (over 1000 parallel workers), invariant to action frequency & delayed…

Dickson Wu

Hi I’m Dickson! I’m an 18-year-old innovator who’s excited to change the course of humanity for the better — using the power of ML!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store