Ladies of Fashion - An Insult At One Time?

I ran across the following book while doing research the other day. Written by H.D. Eastman in 1853, it's called Fast man's directory and lovers' guide to the ladies of fashion and houses of pleasure in New-York and other large cities. What strikes me is the understanding that "ladies of fashion" is a term for prostitutes. I didn't know that! I recall that high end courtesans and prostitutes in England were called "fashionable impures" in the early nineteenth century, but didn't know about this American usage.

Does anyone know the story behind this label and how long it lasted?