Once upon a time, when reading and evaluation fees were the commonest form of upfront fee, the amateur and disreputable agents who charged them justified them by claiming that it takes time to read a manuscript, and an agent shouldn't be expected to do that for free.
Where this argument falls apart: if a manuscript isn't publishable (and most aren't), you don't necessarily have to read it all the way through to know. Much of the time, you don't have to read beyond the first fifty pages. Sometimes, you don't even need to read beyond the first paragraph.
Don't believe it? You're not alone. Plenty of writers refuse to accept that it's possible to evaluate, let alone reject, an entire book-length manuscript (i.e., their manuscript) on the basis of just a few pages (somehow you never hear that argument if the same few pages result in a request to submit, but never mind). It's not that most manuscripts are so bad, they think, it's that agents are lazy. Or prejudiced against new writers. Or cantankerous curmudgeons just looking for excuses to reject, cackling with glee every time they send out a form rejection letter (while hoarding your paper clips and steaming the stamps off your SASEs). This is a common subject of discussion in writers' forums, and the cause of a considerable amount of bitterness.
(There's a similar level of denial about query letters, which many writers resent because they feel that a one-page business letter can't tell an agent or editor anything about the quality of their writing. Not so. I get up to 100 emails a week from writers asking questions or making complaints, and it's often clear to me from reading these letters--ungrammatical, mis-spelled, poorly punctuated, sometimes with malapropisms and homophone errors--that the writer isn't ready to be submitting. Do I mention this? No. That's not my job. But it gives me a lot of insight into the quality of the slush pile.)
If you want to see just how easy it really is to reject some manuscripts, pay a visit to POD-dy Mouth, whose most recent post addresses the issue of how bad a (POD-pubbed) book must be to only read one sentence or paragraph before tossing it aside. There are examples (don't be drinking anything while you're reading them). My favorite one-sentence toss-out: "They called her Labia."
Obviously, not everything that's out there is this bad. Still, the hard truth is that most manuscripts aren't publishable. Should you be depressed? Look at it this way. If your work is publishable, you aren't competing with every other hungry writer with a manuscript to sell--just with the five percent or so (the estimate varies depending on who you ask) who've also written publishable books.
Unfortunately, the only way to know for sure that you're publishable is actually to be published (by a legitimate publisher). And the only way to discover that is to submit. On that basis, we're all in the same boat.