“I can feel the magic in this.”
The Yellow Gem is only found in the Ruins. It is used to craft the Star Caller's Staff and the Magiluminescence.
Gathering
Yellow Gems are obtained by mining Ancient Statues, breaking certain Relics and opening Ornate Chests.
Reign of Giants
In the Reign of Giants DLC there is an alternative method of getting Yellow Gems, by destroying Tumbleweed.