At age 4 1/2, you have at least a year, if not two, before you need to do any formal 'school' with him. So you have time to develop routines and figure out how to work more formal studies in. Both my DD6 and DS (5 next week) have been very early readers, so I know it's tempting to start full blown 'school', but there is really no need to do very much. CM emphasized waiting until at least age 6 before doing anything very structured.
To give you an idea of how little time it can take, here's what my DS5 does for school. He wouldn't even do all of this yet (like the Spanish and the Science) if he didn't have a big sister to tag along with.
Morning Family studies (all children, ages 2, 4, 5, and 6, four days a week): Devotional, sing a hymn, recite scripture memory verses, read a poem, and an Early Years level story or two. Then we alternate a Spanish Video with lessons from 106 Days of Creation. Total Time: less than 45 minutes (slightly more on Spanish days). We also do an artist study for a couple of minutes once a week and listen to composers during lunch, and practicing using our Spanish vocabulary as we eat.
Afternoon work (4 days a week): DS5 practices printing numerals for about 3-5 minutes. This is all he can tolerate. Fine motor skills are NOT his thing, but he is into math, and wants to learn the numbers. We buddy read a story for about 10 minutes (alternating one page each of short chapter books). Then we often play a math card game, or a game like battleship or dominoes, or he has some educational computer games he can play for a little while each day. We also 'talk math' often throughout the day, as this is his thing. Later we read a bedtime story or two, and then DS reads to himself for a while.
On the fifth day we do nature study such as taking a walk in the river valley, or a nature oriented field trip (zoo, nature centre, etc.) He doesn't have the drawing skills to keep a nature journal, so I just have each child find something to 'notice' each week. Other than that we focus on LOTS of time outdoors, learning habits, and helping around the house.
You could easily fit all of that into the schedule you described, especially if your mom is able to help do things like read stories and take a nature walk together. And that would be on top of the preschool, which is all the 'school' most 4 year olds do anyway.
Don't overload yourself or your son! Just ease into it. With my DD6 we started one or two new subjects each month for about 6 months until we had the full program going. She still only does the 45 minutes of family studies in the morning and about 45 minutes more in the afternoon, 4 days a week. It really doesn't take that much time at this level.