I wanted to add that since your oldest is still so young, and the lessons will be short, you can also try “doing school” only during nap times for your youngest. I see you said that you don’t want to do this, but you can save it for an option if you have to.
It definitely is a juggling act, and you have to be flexible but it is doable.
Also, as someone else said, the youngest will probably want to “do school” with your older ones. That’s the way it’s always worked in our house. Get some inexpensive coloring books or just print out some coloring pages from the internet, put them in a binder and let that be your 2yo’s “work”. My youngest just turned 3 in June, and all through our last school year, it was so cute to hear him say that he wanted to “do his math, too” (he wasn’t doing math of course, but he heard the others say it). He He would sit right at the table with us and color or play with the Math blocks.
And be encouraged…the younger ones will pick up so much more from being in the rich environment, that it actually will make your life easier in the long run 
Nanci