Vitaly Vanchurin on Thermodynamics of Machine Learning

41:57 Trying to understand the thermodynamics of neural networks in terms of temperature, entropy, ensemble spaces etc. Then in the next 8 minutes or so he explains how he arrived, via grand canonical ensembles, at a linear dynamics described by a kind of Schrödinger equation.

At 57:00 there is a rather confused discussion about some "second law of learning" but he never seems to get around to actually saying what it is. Maybe it's just that the second law of thermodynamics should be restated as something like "The entropy in a closed system increases at the maximum possible rate".

See also Philip Ball and Stuart Kauffman on Biology.

Subscribe to Curt Jaimungal. 

Comments

Post a Comment

Popular posts from this blog

Steven Johnson - So You Think You Know How to Take Derivatives?

Image
A derivative is linearisation, and differential calculus is essentially linear algebra, ...  See  Freya Holmér - Why Can't You Multiply Vectors?  and  Freya Holmér on Continuity of Splines . See also the MIT OCW page:  Matrix Calculus For Machine Learning And Beyond  (Alan Edelman, Steven G. Johnson) Subscribe to The Julia Programming Language . Alan Edelman talking about expressing mathematics as computer code. The idea is that you can use computer languages to communicate mathematical ideas precisely to other people. See my comments about functional programming languages here:  https://prooftoys.org/ian-grant/hm/ Subscribe to TEDx Talks .
Read more

Hitachi HD44780U LCD Display Fonts

Image
I finally got the bit-mapped fonts from the Hitachi "1602" display :  Here's some JavaScript: const HD44780U01ASCII=[ [],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], [],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ], [4, 4, 4, 4, 0, 0, 4, 0, 0, 0, ], [10, 10, 10, 0, 0, 0, 0, 0, 0, 0, ], [10, 10, 31, 10, 31, 10, 10, 0, 0, 0, ], [4, 15, 20, 14, 5, 30, 4, 0, 0, 0, ], [24, 25, 2, 4, 8, 19, 3, 0, 0, 0, ], [12, 18, 20, 8, 21, 18, 13, 0, 0, 0, ], [12, 4, 8, 0, 0, 0, 0, 0, 0, 0, ], [2, 4, 8, 8, 8, 4, 2, 0, 0, 0, ], [8, 4, 2, 2, 2, 4, 8, 0, 0, 0, ], [0, 4, 21, 14, 21, 4, 0, 0, 0, 0, ], [0, 4, 4, 31, 4, 4, 0, 0, 0, 0, ], [0, 0, 0, 0, 12, 4, 8, 0, 0, 0, ], [0, 0, 0, 31, 0, 0, 0, 0, 0, 0, ], [0, 0, 0, 0, 0, 12, 12, 0, 0, 0, ], [0, 1, 2, 4, 8, 16, 0, 0, 0, 0, ], [14, 17, 19, 21, 25, 17, 14, 0, 0, 0, ], [4, 12, 4, 4, 4, 4, 14, 0, 0, 0, ], [14, 17, 1, 2, 4, 8, 31, 0, 0, 0, ], [31, 2, 4, 2, 1, 17, 14, 0, 0, 0, ], [2, 6, 10, 18, 31, 2, 2, 0, 0,...
Read more

Using Pipewire to Make A Music Synthesizer

Image
Illustration from Project Gutenberg's The Tenniel Illustrations for Carroll's Alice in Wonderland by Sir John Tenniel. Thinking about the linux desktop audio problem: I'm not the only one who finds this weird: https://t.co/b2BZMiak07 — eternal Doorman (@IanANGrant66) May 17, 2025  I noticed that if you change the volume using the Gnome volume keys it changes it in jumps. Usually you don't notice, but if you play a pure sine tone then you can easily hear the distortion. If you want to play with pipewire from C on Ubuntu 24.04 then it's really easy. The pipewire tutorials and examples can be compiled and run if you just first install the pipewire-0.3-dev package with sudo apt install libpipewire-0.3-dev But the API is sooo far away from pluggable, typed streams that I find it quite depressing. Is it because nobody understands the necessary type system . See  Programming Channel Processes and Alice's Adventures In The Wonderland Wide Web . [Update: y...
Read more