Cheval began the building in April 1879. He reports:
"I was walking very fast when my foot caught on something that sent me stumbling a few meters away, I wanted to know the cause. In a dream I had built a palace, a castle or caves, I cannot express it well... I told no one about it for fear of being ridiculed and I felt ridiculous myself. Then fifteen years later, when I had almost forgotten my dream, when I wasn't thinking of it at all, my foot reminded me of it. My foot tripped on a stone that almost made me fall. I wanted to know what it was... It was a stone of such a strange shape that I put it in my pocket to admire it at my ease. The next day, I went back to the same place. I found more stones, even more beautiful, I gathered them together on the spot and was overcome with delight... It's a sandstone shaped by water and hardened by the power of time. It becomes as hard as pebbles. It represents a sculpture so strange that it is impossible for man to imitate, it represents any kind of animal, any kind of caricature."
"I said to myself: since Nature is willing to do the sculpture, I will do the masonry and the architecture"
For the next thirty-three years, Cheval picked up stones during his daily mail round and carried them home to build the Palais idéal. He spent the first twenty years building the outer walls. At first, he carried the stones in his pockets, then switched to a basket. Eventually, he used a wheelbarrow. He often worked at night, by the light of an oil lamp.
The Palais is a mix of different styles with inspirations from Christianity to Hinduism. Cheval bound the stones together with lime, mortar and cement.