The authors base their conclusion on a close physical resemblance between orangutans and humans, which they say has been overshadowed by genetic evidence linking us to chimps.
What's more, the study authors argue, the genetic evidence itself is flawed. (Get a genetics overview.)
John Grehan, of the Buffalo Museum of Science in New York State, and Jeffrey Schwartz, of the University of Pittsburgh in Pennsylvania, say that the DNA evidence cited by many scientists only looks at a small percentage of the human and chimp genomes.
What's more, the genetic similarities likely include many ancient DNA traits that are shared across a much broader group of animals.
By contrast, humans share at least 28 unique physical characteristics with orangutans but only 2 with chimps and 7 with gorillas, the authors say.
The finding, which has the potential to spark a radical rethink of human origins, is being met with caution.
"There are many paleontologists and molecular biologists who are heaping scorn on this paper," noted Peter Andrews of the Natural History Museum in London.
Even though he still backs the human-chimp relationship, Andrews had recommended that the study be published, and it now appears in the June issue of the Journal of Biogeography.
"It is controversial," he said, "but I think it is a subject that needed to be aired."
With the sequencing of the chimpanzee genome in 2005, scientists found direct proof that humans and chimps are 96 percent the same genetically.
But looking at physical traits rather than genetic ones, orangutans are a better match, Grehan and Schwartz say.
Tell-tale features shared by both orangutans and humans include thickly enameled molar teeth with flat surfaces, greater asymmetries between the left and right side of the brain, an increased cartilage-to-bone ratio in the forearm, and similarly shaped shoulder blades.
"A hole in the roof of the mouth that was supposedly unique to humans is also present in orangs," Schwartz said.
"Humans and orangs have the widest-separated mammary glands, and they grow the longest hair," he added. "Humans and orangs actually have a hairline, in contrast to virtually all primates, where the hair comes down to the top of the eyes."
The team also highlighted orangutan-type traits in the teeth and jaw remains of ancient fossil apes from Africa and Europe.
Based on their analysis, the authors suggest "that humans and orangutans share a common ancestor that excludes [living] African apes."
But orangutans are native to Southeast Asia, which creates a problem: How did humans evolve in Africa if we are so closely related to the geographically distant orangutan? (Explore a human migration time line.)
The mainstream view is that humans evolved from the same group as African great apes: chimps, bonobos, and gorillas.
Instead, the authors speculate that a widely distributed orangutan-like ancestor of humans lived in Africa, Europe, and Asia some 13 million years ago.
Subsequent changes in climate and environment likely caused many populations to become extinct, leaving Asian and African species to evolve in isolation.
"There are actually very few [physical] features linking chimps and humans," noted the Natural History Museum's Andrews. "The case for that is based almost entirely on molecular evidence."
And those molecular studies are flawed, Schwartz and Grehan say, because of the high likelihood that the data includes broadly shared DNA traits.
"When you're doing a really rigorous analysis of relationships, you don't just stop at the potential demonstration of similarity," Schwartz said. "You have to distinguish between features that are widely shared [among many species] and those that are more uniquely shared."
In addition, Schwartz notes, the most cited studies are largely based on the so-called coding region of the genome, which makes up just 2 to 3 percent of an animal's DNA.
Scientists are referring to this tiny part of the genome when they say humans and chimps are so similar, he said.
But other studies that focus on non-coding regions also consistently support a human-chimp link, counters Carel van Schaik of the Anthropological Institute and Museum at the University of Zurich, Switzerland.
"A study that reaches a very different conclusion [from the genetic evidence] must explain why these molecular studies are wrong," van Schaik, who also serves as a consultant to the conservation group Borneo Orangutan Survival UK, said in an email.
"Of course, orangutans are very human-like in many respects, but so are chimpanzees, bonobos and gorillas … ."
Anthropologist Nick Newton-Fisher, of the University of Kent in the U.K., described the human evolutionary path implied by the new study as a "wacky idea."
"Given the weight of evidence from the genetics," he said, he would be reluctant to accept the new findings.