To find the Gibbering Gemstone, a piece required to craft the Staff of Herding and unlock Whimsyshire, you must go to Act III at the Fields of Slaughter (quest is called, 'Destroy the Ballistae'). Find a cave called 'Cavern of Frost'. Sometimes the cave may be called Icefall Caverns, and this means you must leave the game and hunt again for the cave that will be named as "Caverns of Frost".
Proceed through level 1 of the Caverns of Frost until you enter Level 2. Somewhere in this second level, you will fight a yellow elite named "Chiltara". Defeating Chiltara will earn you that elusive loot called the Gibbering Gemstone.
if Chiltara is not in the Caverns of Frost level 2 and there is another named Elite like Drax, etc, repeat the entire process from the start until u eventually meet Chiltara in place of that randomized elites. It took me about 20 tries.