There is a need to clarify the relationship between traditional
symbolic computation and neural network computation. We suggest that
traditional context-free grammars are best understood as a special
case of neural network computation; the special case derives its power
from the presence of certain kinds of symmetries in the weight
values. We describe a simple class of stochastic neural networks,
Linear Dynamical Automata (LDAs), define Lyapunov Exponents for these
networks, and show that they exhibit a significant range of dynamical
behaviors contractive and chaotic, with context free grammars at the
boundary between these regimes. Placing context-free languages in this
more general context has allowed us to make headway on the challenging
problem of designing neural mechanisms that can learn them.