## Self-Supervised Learning: a suvey

Self-Supervised Representation Learning Broadly speaking, all the generative models can be considered as self-supervised, but with different goals: Generative models focus on creating diverse and realistic images While self-supervised representation learning care about producing good features generally helpful for many tasks Image based Distortion Exemplar-CNN (Dosovitskiy et al., 2015) Rotation of an entire image (Gidaris et al.

## Loss functioin in neural network

Kullback-Leibler divergence Information theory Quantify information of intuition1 Likely events should have low information content Less likely events should have higher information content Independent events should have additive information. For example, finding out that a tossed coin has come up as heads twice should convey twice as much information as finding out that a tossed coin has come…

## 炼丹心法

What is the difference between an autoencoder and an encoder decoder? 参数 神经网络的调参顺序? 神经网络中的Epoch、Iteration、Batchsize 神经网络…

## Pytorch 小技巧

Tensor Torch 的广播机制 Contigious vs non-contigious tensor Optimizer 如果使用 cuda，那么要先构造 optimizers 再创建 model，参考官方文档。 Effect of calling model.cuda() after constructing an optimizer 查看参数个数…