Gems can be found randomly in the forest during "special events." You could only recieve 1 gem from "finding" one, but you could get 10 gems from mining. Gems can also be found just as turns would, in stores/shops. There is almost always a chance to get gems where you can get turns. Places like the outhouse offer you a chance to get a gem as well.
Gems are rare and are not easy to find. Collecting gems takes time as they are very valuable for buying mounts, hitpoints, and a few other things.