In the first place, family trees are more than a little hypothetical.  Nobody really knows for sure who all of his distant male ancestors really were.  All we have is the word of the mothers who were involved and that's notoriously unreliable.  In that regard, many people don't even know the identities of their actual fathers.  They think that they do, but maybe they really don't.  Even if what the mothers had to say was always completely honest and accurate, then we'd still go wrong a lot of the time because the further into the past we trace a family tree the less reliable the records become.  What with one thing and another, there are even situations in the ancestry of every individual that cause the identities of even some of the female ancestors to be a mystery.  However, there's another even better reason to be scornful of boasts about ancient lineage.  That reason is simple arithmetic.

Let's assume that each generation of human ancestry takes about 25 years.  It's a bold assumption but 25 years is probably as good as any other number and it makes the arithmetic easier.  For the sake of convenience, let's designate me (or yourself) as being of the generation of 1950.  I was alive in 1950.  If you weren't, then pick your own decade and do your own calculations.  The difference, I assure you, will be insignificant.  We must also assume that the number of my (or your) ancestors will double with each generation into the past.  That is, you had two parents, four grandparents, eight great-grandparents, and so forth.  Social scruples relating to consanguinity will usually make that a safe assumption for several generations into the past.  Beyond those few generations, it isn't a good assumption any more.  That is, the further back into the past you trace your family tree, the more likely it is that some particular individual might occupy more than one place in that family tree.  So, the actual number of ancestors that you have at some point in the past will probably be smaller than the number predicted by my simple-minded doubling method.  Nevertheless, the fact remains that the number of ancestors will be large.

In the accompanying table, I've shown the theoretically maximum number of ancestors that I had at different times in the past.  I've also shown the estimated population of the world at different times in the past.  The numbers of my ancestors are based on the assumptions that I've mentioned above.  The estimates of world population are from the U.S. Census Bureau.  The numbers of my ancestors are approximate but they're probably every bit as accurate as the numbers provided by the Census Bureau.

You can see that if you trace your family tree back to even as recent a time as the 1200's (less than a thousand years, an eye-blink in the history of the human race!) then it's pointless to worry about who your ancestors were.  The theoretical possible number of ancestors at that time was almost three times the population of the entire world at that time.  In theory, you could be descended from every human being who was alive on the planet as few as 800 years ago.

I suppose that the lesson here is that the only ancestors who really matter are the ones that you can actually remember for yourself.  Beyond that, they're irrelevant.

---

Reference
U. S. Census Bureau, Historical Estimates of World Population, http://www.census.gov/ipc/www/worldhis.html