I don’t think you should worry about the number of books, just think about it as a library your children can access daily.
I began logging all my kids’ books when my oldest was about 6 mo. old. We had almost 2K toddler-to-early elementary books; since learning about CM, I’ve eliminated about 600 I consider twaddle. Some were from my childhood, some we purchased, but most were hand-me-downs from the 9 older cousins. We have read almost all the books, including the twaddle, to my DS(3), and he loves everything. He’ll have a favorite for a week or two – sometimes three, then he’ll pull a different set of books out and decide they’re his new favorites. My other DS(21mo) is starting to really appreciate the books too! He also has his favorites that change a little over time. They both love to look through their bookcase and pull out a new or forgotten book to be read anew.
There doesn’t seem to be any ill effect from the massive reading material (they still spend 3-4 hours outside daily). I think the varied reading has increased their attention spans, language skills, and active vocabularies. We read to DS(3) about an hour every night and he’d keep going if his eyes would stay open; his active vocabulary is around 6K words, he uses complex sentences, and averages 10 words per sentence. We read to DS(21mo) about 30 min. per day. His active vocabulary is over 1K words and speaks as well as most 3 year olds. We are even able to read some pretty complicated books to both now. We just finished James Herriot’s Treasury for Children; the younger still nurses and loved hearing about the baby animals nursing from their mothers.