Many people treat autoencoders as data compression tools—like using a Ferrari for grocery shopping. Technically not wrong, but entirely missing the point.


In Chapter 5 of the new book, we delve into the essence of (compression-based) autoencoders: their core value lies in ensuring consistency and self-consistency in structural-revealing representations. This concept is far older and deeper than most realize.

Interestingly, the comments reveal typical misconceptions:

- Some boast about classifying MNIST with 10-dimensional encodings (which actually proves the structure-preserving capability of representations)

- Others ask when the book will be available (soon, but Chapter 5 still needs major revisions)

- A few mention work on sparse coding in overcomplete dictionaries using extended autoencoders (indeed relevant—see the paper)


The most ironic moment? When JC Wong pointed out the discontinuity issues in autoencoders for generative purposes, it perfectly reinforced our argument: without maintaining structural consistency in representation space, even the fanciest applications are castles in the air.

Related paper: [Sparse Coding and Autoencoders](https://ieeexplore.ieee.org/document/8437533)

(I’m quite happy with the chapter’s illustrations, but the text needs more polish—after all, distilling 20 years of research without sounding like a textbook cliché is no easy feat.)