Resources
Here are some great resources for further exploring related topics. 📚 stands for a book, 📝 for blog posts / papers. I tried to assign each to one of the three parts of this mini-course, but books are often relevant for several parts.
Part I: Bayes' rule, KL divergence, (cross-)entropy
- 📝 Bayes' rule by alexei and E. Yudkowsky
- 📝 Six and a half intuitions for KL divergence by CallumMcDougall.
- 📚 Probability Theory: The Logic of Science by E. Jaynes 1
- 📚 Entropic Physics by Ariel Caticha
- 📚 Information Theory, Inference, and Learning Algorithms by David MacKay
- 📝 Intuition behind KL Divergence by Johannes Schusterbauer.
Part II: Optimization
- 📝 Why KL?, KL is all you need, diffusion models by Alex Alemi.
- 📝 Maximum Entropy and Bayes' rule by Rising Entropy
- 📚 Machine Learning: A Probabilistic Perspective by Kevin P. Murphy
- 📝 The principle of maximum entropy by Casey Chu
Part III: Compression
- 📚 Elements of Information Theory by Thomas Cover and Joy Thomas
- 📝 Why Philosophers Should Care About Computational Complexity by Scott Aaronson