
(Note: this section is based largely on the following material: the documentation ("A Little bit of history" from the computer program FRACTINT; James Gleick, Chaos, Making of a New Science; and Ian Stewart, Does God Play Dice?  the Mathematics of Chaos).
Up until the nineteenth century, the vision of Copernicus and Brahe, Kepler and Newton was that nature  such as the paths of planets, comets, and projectiles  could be described in ellipses, parabolas, and hyperbolas. Mathematics and geometry was based on "wellbehaved" formulas that worked well in modeling nature.
But beginning in the early 1870s, a 50year crisis transformed mathematical thinking. Cantor showed how a simple, repeated procedure could turn a line into a dust of scattered points (the socalled Cantor Set). Peano generated a convoluted curve that eventually touches every point on a plane, creating shapes that fall between the usual categories of onedimensional lines, twodimensional planes and threedimensional volumes. Poincare attempted to analyze the stability of the solar system in the 1880s and found that the problem of the interrelation of the many planets resisted conventional methods of calcultion. Instead, he developed a qualitative approach, a "state space" in which each point represented a different planetary orbit, and studied the topology  the shape or connectedness  of whole families of orbits. This revealed that while many initial orbits quickly settled into the familiar curves, there were also strange, "chaotic" orbits that never became periodic and predictable. Also resisting the traditional mathematical models were fluctuating, random phenomena such as the flooding of the Nile, price series in economics, and the jiggling of molecules in Brownian motion in fluids.
Does God Play Dice? : The Mathematics of Chaos by Ian Stewart
This mesmerizing historical overview of nonlinear science, full of seedy ideas and fascinating expositions (from heartbeat to weather forecast) is well worth reading. One of those "aha !" books that will broaden your understanding of the universe (and the rest), it is very "visual" and..well, a friend of mine said she considered it
a "mental thriller" since it touches on the great old questions of determinism and predictability. As for "mathematics" in the title don't be put off. The book is virtually mathless.
rating:
accessibility  Nontechnical (Popular science level)
quality 
Synergetics : an introduction : nonequilibrium phase transitions and selforganization in physics, chemistry, and biology by Hermann Haken (out of print)
This is a wonderful interdisciplinary book, covering statistical physics and thermodynamics, chaos, economics, biology,...If you want to know what buzzwords like complexity, selforganization, chaotic behavior and old ones that have gained new lustre, like entropy and information, mean you should consult this masterful introduction. For prosother books in the Springer Series on Synergetics are indispensable. For the layman this one will suffice to trigger your own imagination.
rating:
accessibility  Intermediate (entry level textbook equivalent)
quality 