Semester | Mand. | Option | Filières |
x |
Lecture | 10 |
Project | 15 |
Study plan | Semester | Mand. | Option | Filières |
EDNE - Neuroscience 2021-22 | x |
Cursus | Sem. | Type |
---|---|---|
Neuroscience | Opt. |
Loss Landscapes of Neural Networks:
In practical applications, Deep Neural Networks are typically trained by walking down the loss surface using gradient descent augmented with a bag of tricks. One of the important practical insights has been that large, over parameterized, networks that have more parameters than necessary work better - one potential interpretation (but not the only one) is the 'lottery ticket hypothesis'. Obviously, the shape of the
loss landscape is important when walking down. In recent years, research on the shape of the loss landscape has addressed questions such as: "Is there one big global minimum or many scattered small ones? ", "Is the loss landscape rough or smooth?", "Should we worry about saddle points?", "Are there flat regions in the loss"?, "How many saddle points are there?". While these look like questions for theoreticians, their answers might have practical consequences and lead to a better understanding of the role of over parameterization, pruning, and reasons of the bag of tricks. The aim of this workshop is to bring together researchers that have worked on these topics from different points of views and different backgrounds (Computer Science, Math, Physics, Computational Neuroscience), and build a community around these questions.
II. WORK LOAD of the students (1 credit = 25-30 hours of work)
A) 10 speakers over 2 days + discussion session (program attached). = 11 hours of work
B) Report preparation for course credit (14 hours of work)
1. Each participating student has to ask at least 2 questions for two different lectures
2. Each student has to submit a 5-page report
page 1: Select the best talk
page 2: - For the same talk
page 3/4: Select second best talk. Answer same points as page 1/2
page 5: Select a bad talk
By the end of this course you should be able to explain recent results on the shape and convergence properties of the loss landscape in neural networks.
Artificial Neural Networks, Deep Learning, Optimization, Gradient Descent, Loss landscape, Convergence, Convexity, Permutation Symmetry
A Master-level class on Artificial Neural Networks or Machine Learning or Optimization
optionnel
optionnel
Cours
Lecture
Heure(s)
Hour(s)
H
H
Projet
Project
Heure(s)
Hour(s)
H
H
Mémoire
Term paper
Session libre
Session free
LIB
LIB
BALISES_TEXTES
BOOLEAN_Symbol
BaliseJS_debut
BaliseJS_fin
Balise_debut
Balise_fin
FO_debut
FO_fin
HTML_debut
HTML_fin
JS_debut
JS_fin
BALISE_BOLD
BOOLEAN_Symbol
BaliseJS_debut
\x5bb\x5d
BaliseJS_fin
\x5b\x2fb\x5d
Balise_debut
[b]
Balise_fin
[/b]
FO_debut
font-weight=bold
FO_fin
font-weight=normal
HTML_debut
<b>
HTML_fin
</b>
JS_debut
\x3cb\x3e
JS_fin
\x3c\x2fb\x3e
BALISE_ITALIC
BOOLEAN_Symbol
BaliseJS_debut
\x5bi\x5d
BaliseJS_fin
\x5b\x2fi\x5d
Balise_debut
[i]
Balise_fin
[/i]
FO_debut
font-style=italic
FO_fin
font-style=normal
HTML_debut
<i>
HTML_fin
</i>
JS_debut
\x3ci\x3e
JS_fin
\x3c\x2fi\x3e
BALISE_RED
BOOLEAN_Symbol
BaliseJS_debut
\x5bred\x5d
BaliseJS_fin
\x5b\x2fred\x5d
Balise_debut
[red]
Balise_fin
[/red]
FO_debut
color=#990033
FO_fin
color=black
HTML_debut
<span style="color:#990033;">
HTML_fin
</span>
JS_debut
\x3cspan style="color:#990033;"\x3e
JS_fin
\x3c\x2fspan\x3e
BALISE_TAB
BOOLEAN_Symbol
BaliseJS_debut
\x5btab\x5d
BaliseJS_fin
Balise_debut
[tab]
Balise_fin
FO_debut
[inline]=        
FO_fin
HTML_debut
<img src="/images/gestacplus/base/pixel.gif" width="40" height="1"/>
HTML_fin
JS_debut
\x3cimg src="\x2fimages\x2fgestacplus\x2fbase\x2fpixel.gif" width="40" height="1"\x2f\x3e
JS_fin
BALISE_LIST
BOOLEAN_Symbol
BaliseJS_debut
\x5bli\x5d
BaliseJS_fin
Balise_debut
[li]
Balise_fin
FO_debut
[inline]=•  
FO_fin
HTML_debut
•  
HTML_fin
JS_debut
\x26#x2022;\x26#160;\x26#160;
JS_fin
CODE5045487
BOOLEAN_Symbol
BaliseJS_debut
\x5b\x5d
BaliseJS_fin
Balise_debut
[]
Balise_fin
FO_debut
[inline]= 
FO_fin
HTML_debut
 
HTML_fin
JS_debut
\x26#160;
JS_fin
BALISE_GREEN
BOOLEAN_Symbol
BaliseJS_debut
\x5bgreen\x5d
BaliseJS_fin
\x5b\x2fgreen\x5d
Balise_debut
[green]
Balise_fin
[/green]
FO_debut
color=#008000
FO_fin
color=black
HTML_debut
<span style="color:#008000;">
HTML_fin
</span>
JS_debut
\x3cspan style="color:#008000;"\x3e
JS_fin
\x3c\x2fspan\x3e
CODE7508550
BOOLEAN_Symbol
BaliseJS_debut
\x5bblue\x5d
BaliseJS_fin
\x5b\x2fblue\x5d
Balise_debut
[blue]
Balise_fin
[/blue]
FO_debut
color=#0080FF
FO_fin
color=black
HTML_debut
<span style="color:#0080FF;">
HTML_fin
</span>
JS_debut
\x3cspan style="color:#0080FF;"\x3e
JS_fin
\x3c\x2fspan\x3e
BALISE_COURIER
BOOLEAN_Symbol
BaliseJS_debut
\x5bcourier\x5d
BaliseJS_fin
\x5b\x2fcourier\x5d
Balise_debut
[courier]
Balise_fin
[/courier]
FO_debut
font-family=Courier
FO_fin
font-family=
HTML_debut
<font style="font-family:Courier;">
HTML_fin
</font>
JS_debut
\x3cfont style="font-family:Courier;"\x3e
JS_fin
\x3c\x2ffont\x3e
BALISE_SIZE20
BOOLEAN_Symbol
BaliseJS_debut
\x5b20\x5d
BaliseJS_fin
\x5b\x2f20\x5d
Balise_debut
[20]
Balise_fin
[/20]
FO_debut
font-size=20pt
FO_fin
font-size=
HTML_debut
<font style="font-size:20pt;">
HTML_fin
</font>
JS_debut
\x3cfont style="font-size:20pt;"\x3e
JS_fin
\x3c\x2ffont\x3e
BALISE_SIZE16
BOOLEAN_Symbol
BaliseJS_debut
\x5b16\x5d
BaliseJS_fin
\x5b\x2f16\x5d
Balise_debut
[16]
Balise_fin
[/16]
FO_debut
font-size=16pt
FO_fin
font-size=
HTML_debut
<font style="font-size:16pt;">
HTML_fin
</font>
JS_debut
\x3cfont style="font-size:16pt;"\x3e
JS_fin
\x3c\x2ffont\x3e
BALISE_SIZE12
BOOLEAN_Symbol
BaliseJS_debut
\x5b12\x5d
BaliseJS_fin
\x5b\x2f12\x5d
Balise_debut
[12]
Balise_fin
[/12]
FO_debut
font-size=12pt
FO_fin
font-size=
HTML_debut
<font style="font-size:12pt;">
HTML_fin
</font>
JS_debut
\x3cfont style="font-size:12pt;"\x3e
JS_fin
\x3c\x2ffont\x3e
BALISE_SIZE8
BOOLEAN_Symbol
BaliseJS_debut
\x5b8\x5d
BaliseJS_fin
\x5b\x2f8\x5d
Balise_debut
[8]
Balise_fin
[/8]
FO_debut
font-size=8pt
FO_fin
font-size=
HTML_debut
<font style="font-size:8pt;">
HTML_fin
</font>
JS_debut
\x3cfont style="font-size:8pt;"\x3e
JS_fin
\x3c\x2ffont\x3e
BALISE_BLACKBACK
BOOLEAN_Symbol
BaliseJS_debut
\x5bblackback\x5d
BaliseJS_fin
\x5b\x2fblackback\x5d
Balise_debut
[blackback]
Balise_fin
[/blackback]
FO_debut
background-color=black
FO_fin
background-color=white
HTML_debut
<span style="background-color:black;">
HTML_fin
</span>
JS_debut
\x3cspan style="background-color:black;"\x3e
JS_fin
\x3c\x2fspan\x3e
BALISE_WHITE
BOOLEAN_Symbol
BaliseJS_debut
\x5bwhite\x5d
BaliseJS_fin
\x5b\x2fwhite\x5d
Balise_debut
[white]
Balise_fin
[/white]
FO_debut
color=white
FO_fin
color=black
HTML_debut
<span style="color:white;">
HTML_fin
</span>
JS_debut
\x3cspan style="color:white;"\x3e
JS_fin
\x3c\x2fspan\x3e
CODE47534071
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bbeta\x5d
BaliseJS_fin
Balise_debut
[beta]
Balise_fin
FO_debut
[inline]=β
FO_fin
HTML_debut
β
HTML_fin
JS_debut
\x26#946;
JS_fin
CODE47534435
BOOLEAN_Symbol
1
BaliseJS_debut
\x5balpha\x5d
BaliseJS_fin
Balise_debut
[alpha]
Balise_fin
FO_debut
[inline]=α
FO_fin
HTML_debut
α
HTML_fin
JS_debut
\x26#945;
JS_fin
CODE47551722
BOOLEAN_Symbol
BaliseJS_debut
\x5bsub\x5d
BaliseJS_fin
\x5b\x2fsub\x5d
Balise_debut
[sub]
Balise_fin
[/sub]
FO_debut
vertical-align=sub
FO_fin
vertical-align=baseline
HTML_debut
<sub>
HTML_fin
</sub>
JS_debut
\x3csub\x3e
JS_fin
\x3c\x2fsub\x3e
CODE48535424
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bLAMBDA\x5d
BaliseJS_fin
Balise_debut
[LAMBDA]
Balise_fin
FO_debut
[inline]=Λ
FO_fin
HTML_debut
Λ
HTML_fin
JS_debut
\x26#923;
JS_fin
CODE48535666
BOOLEAN_Symbol
1
BaliseJS_debut
\x5blambda\x5d
BaliseJS_fin
Balise_debut
[lambda]
Balise_fin
FO_debut
[inline]=λ
FO_fin
HTML_debut
λ
HTML_fin
JS_debut
\x26#955;
JS_fin
CODE47551793
BOOLEAN_Symbol
BaliseJS_debut
\x5bsup\x5d
BaliseJS_fin
\x5b\x2fsup\x5d
Balise_debut
[sup]
Balise_fin
[/sup]
FO_debut
vertical-align=super
FO_fin
vertical-align=baseline
HTML_debut
<sup>
HTML_fin
</sup>
JS_debut
\x3csup\x3e
JS_fin
\x3c\x2fsup\x3e
CODE56031535
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bdelta\x5d
BaliseJS_fin
Balise_debut
[delta]
Balise_fin
FO_debut
[inline]=δ
FO_fin
HTML_debut
δ
HTML_fin
JS_debut
\x26#948;
JS_fin
CODE56073309
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bpi\x5d
BaliseJS_fin
Balise_debut
[pi]
Balise_fin
FO_debut
[inline]=π
FO_fin
HTML_debut
π
HTML_fin
JS_debut
\x26#960;
JS_fin
CODE56074095
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bDELTA\x5d
BaliseJS_fin
Balise_debut
[DELTA]
Balise_fin
FO_debut
[inline]=Δ
FO_fin
HTML_debut
Δ
HTML_fin
JS_debut
\x26#916;
JS_fin
CODE56074172
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bMU\x5d
BaliseJS_fin
Balise_debut
[MU]
Balise_fin
FO_debut
[inline]=Μ
FO_fin
HTML_debut
Μ
HTML_fin
JS_debut
\x26#924;
JS_fin
CODE56074264
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bPI\x5d
BaliseJS_fin
Balise_debut
[PI]
Balise_fin
FO_debut
[inline]=Π
FO_fin
HTML_debut
Π
HTML_fin
JS_debut
\x26#928;
JS_fin
CODE56074328
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bSIGMA\x5d
BaliseJS_fin
Balise_debut
[SIGMA]
Balise_fin
FO_debut
[inline]=Σ
FO_fin
HTML_debut
Σ
HTML_fin
JS_debut
\x26#931;
JS_fin
CODE56075056
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bALPHA\x5d
BaliseJS_fin
Balise_debut
[ALPHA]
Balise_fin
FO_debut
[inline]=Α
FO_fin
HTML_debut
Α
HTML_fin
JS_debut
\x26#913;
JS_fin
CODE56075186
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bBETA\x5d
BaliseJS_fin
Balise_debut
[BETA]
Balise_fin
FO_debut
[inline]=Β
FO_fin
HTML_debut
Β
HTML_fin
JS_debut
\x26#914;
JS_fin
BALISE_SYMBOL
BOOLEAN_Symbol
BaliseJS_debut
\x5bsymbol\x5d
BaliseJS_fin
\x5b\x2fsymbol\x5d
Balise_debut
[symbol]
Balise_fin
[/symbol]
FO_debut
font-family=Symbol
FO_fin
font-family=
HTML_debut
<font style="font-family:Symbol;">
HTML_fin
</font>
JS_debut
\x3cfont style="font-family:Symbol;"\x3e
JS_fin
\x3c\x2ffont\x3e
CODE56072935
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bmu\x5d
BaliseJS_fin
Balise_debut
[mu]
Balise_fin
FO_debut
[inline]=μ
FO_fin
HTML_debut
μ
HTML_fin
JS_debut
\x26#956;
JS_fin
CODE56073842
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bsigma\x5d
BaliseJS_fin
Balise_debut
[sigma]
Balise_fin
FO_debut
[inline]=σ
FO_fin
HTML_debut
σ
HTML_fin
JS_debut
\x26#963;
JS_fin
CODE68199232
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bphi\x5d
BaliseJS_fin
Balise_debut
[phi]
Balise_fin
FO_debut
[inline]=φ
FO_fin
HTML_debut
φ
HTML_fin
JS_debut
\x26#966;
JS_fin
CODE68199285
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bPHI\x5d
BaliseJS_fin
Balise_debut
[PHI]
Balise_fin
FO_debut
[inline]=Φ
FO_fin
HTML_debut
Φ
HTML_fin
JS_debut
\x26#934;
JS_fin
CODE68199991
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bgamma\x5d
BaliseJS_fin
Balise_debut
[gamma]
Balise_fin
FO_debut
[inline]=γ
FO_fin
HTML_debut
γ
HTML_fin
JS_debut
\x26#947;
JS_fin
CODE68200031
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bGAMMA\x5d
BaliseJS_fin
Balise_debut
[GAMMA]
Balise_fin
FO_debut
[inline]=Γ
FO_fin
HTML_debut
Γ
HTML_fin
JS_debut
\x26#915;
JS_fin
CODE68200077
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bomega\x5d
BaliseJS_fin
Balise_debut
[omega]
Balise_fin
FO_debut
[inline]=ω
FO_fin
HTML_debut
ω
HTML_fin
JS_debut
\x26#969;
JS_fin
CODE68200150
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bOMEGA\x5d
BaliseJS_fin
Balise_debut
[OMEGA]
Balise_fin
FO_debut
[inline]=Ω
FO_fin
HTML_debut
Ω
HTML_fin
JS_debut
\x26#937;
JS_fin
CODE72092691
BOOLEAN_Symbol
BaliseJS_debut
\x5beta\x5d
BaliseJS_fin
Balise_debut
[eta]
Balise_fin
FO_debut
[inline]=η
FO_fin
HTML_debut
η
HTML_fin
JS_debut
\x26#951;
JS_fin
CODE72092805
BOOLEAN_Symbol
BaliseJS_debut
\x5bETA\x5d
BaliseJS_fin
Balise_debut
[ETA]
Balise_fin
FO_debut
[inline]=Η
FO_fin
HTML_debut
Η
HTML_fin
JS_debut
\x26#919;
JS_fin
BALISE_SOULIGNER
BOOLEAN_Symbol
BaliseJS_debut
\x5bu\x5d
BaliseJS_fin
\x5b\x2fu\x5d
Balise_debut
[u]
Balise_fin
[/u]
FO_debut
text-decoration=underline
FO_fin
text-decoration=none
HTML_debut
<u>
HTML_fin
</u>
JS_debut
\x3cu\x3e
JS_fin
\x3c\x2fu\x3e
CODE72092388
BOOLEAN_Symbol
BaliseJS_debut
\x5bpsi\x5d
BaliseJS_fin
Balise_debut
[psi]
Balise_fin
FO_debut
[inline]=ψ
FO_fin
HTML_debut
ψ
HTML_fin
JS_debut
\x26#968;
JS_fin
CODE72092541
BOOLEAN_Symbol
BaliseJS_debut
\x5bPSI\x5d
BaliseJS_fin
Balise_debut
[PSI]
Balise_fin
FO_debut
[inline]=Ψ
FO_fin
HTML_debut
Ψ
HTML_fin
JS_debut
\x26#936;
JS_fin
CODE140502511
BOOLEAN_Symbol
1
BaliseJS_debut
\x5brho\x5d
BaliseJS_fin
Balise_debut
[rho]
Balise_fin
FO_debut
[inline]=ρ
FO_fin
HTML_debut
ρ
HTML_fin
JS_debut
\x26#961;
JS_fin
CODE140504636
BOOLEAN_Symbol
1
BaliseJS_debut
\x5bRHO\x5d
BaliseJS_fin
Balise_debut
[RHO]
Balise_fin
FO_debut
[inline]=Ρ
FO_fin
HTML_debut
Ρ
HTML_fin
JS_debut
\x26#929;
JS_fin
CODE148292791
BOOLEAN_Symbol
BaliseJS_debut
\x5bepsilon\x5d
BaliseJS_fin
Balise_debut
[epsilon]
Balise_fin
FO_debut
[inline]=ε
FO_fin
HTML_debut
ε
HTML_fin
JS_debut
\x26#949;
JS_fin
CODE169976220
BOOLEAN_Symbol
BaliseJS_debut
\x5blig\x5d
BaliseJS_fin
Balise_debut
[lig]
Balise_fin
FO_debut
[inline]=›  
FO_fin
HTML_debut
›  
HTML_fin
JS_debut
\x26#x203A;\x26#160;\x26#160;
JS_fin
CODE148292899
BOOLEAN_Symbol
BaliseJS_debut
\x5bEPSILON\x5d
BaliseJS_fin
Balise_debut
[EPSILON]
Balise_fin
FO_debut
[inline]=Ε
FO_fin
HTML_debut
Ε
HTML_fin
JS_debut
\x26#917;
JS_fin
BALISETXT_SAUT
BOOLEAN_Symbol
BaliseJS_debut
\x5bbr\x2f\x5d
BaliseJS_fin
Balise_debut
[br/]
Balise_fin
FO_debut
FO_fin
HTML_debut
<br/>
HTML_fin
JS_debut
\x3cbr\x2f\x3e
JS_fin