The remains of a prehistoric primate that lived high in the Andes 20 million years ago and was so small it could fit in your hand is helping scientists learn more about how human brains evolved.

In a study published Wednesday in the journal Science Advances, researchers in China and the U.S. used high resolution imaging to examine the only known fossilized skull of the extinct Chilecebus, a New World monkey that scampered around ancient mountain forests, feeding on leaves and fruit.

One key finding: the brain size of primates, long assumed to have increased progressively over time, now appears to have followed a more roundabout path.

Primates are broadly split into two groups: Old World, from which our own species descended, and New World species of the Americas and Oceania.

"We see multiple episodes of expansion of the brain in each of these major groups, and we see several episodes of actual reduction of relative brain size in certain groups," co-author John Flynn of the American Museum of Natural History told AFP.

The research, led by Ni Xijun, at the Chinese Academy of Sciences, used X-rays and CT scans of the inside of the Chilebus' skull to determine its internal structure.

The fossilized skull has been precisely dated because it was discovered among volcanic rocks, and by placing the species within its bigger family tree, the team was able to infer that cerebral enlargement occurred repeatedly and independently in anthropoid evolution.

Though the Chilecebus was roughly around the size of a modern marmoset or tamarin, in contrast to those monkeys its brain had several grooves known as infolding that suggests a greater degree of cognitive complexity: in other words, brain size is not always linked to advancement.

What's more, in modern primates, the size of the visual and olfactory centers of the brain are inversely related, meaning that species with a strong sense of sight usually have a weaker sense of smell and vice versa.

But the researchers found that a small olfactory bulb in Chilecebus did not result in a stronger olfactory ability, meaning the two abilities aren't coupled as previously thought.

Flynn said the research bore testament to the secrets that could be unlocked from well-preserved ancient fossils.

"We can go out into the mountains and make this remarkable discovery 10,000 feet up in the Andes and be able to make insights about the evolution of our history, be able to test previous hypotheses… (and) be able to understand the evolution of brain complexity in primates.

"That's a really amazing possibility from the discovery of one really well preserved fossil."