I started the Primer with DD when she was 4. We took a very laid back approach, doing it when she was interested. She went to Christian school for kindergarten, but she was still enjoying the Primer, even asking to do math when she was on break or over the weekends. She was actually learning more from it than the Saxon math they were doing at school.
She had a little left from the Primer when we started homeschooling for 1st grade. When we started Alpha I had her test out on the repeated lessons and we moved on. She is now almost finished with Alpha.
DS has recently started the Primer. He asks to do math with his sister, but since he is only in preschool, like with his sister, I just let him do it when he asks for it.
So, is the Primer worth it? For us it is. My kids love math and enjoy the blocks. But if you decide to not go through the Primer your child is going to be fine starting in Alpha in 1st grade.