We definitely have some kind of activities for the little ones, though it's not really structured. I think a big thing at this age is to begin getting them to stick with doing the same thing for longer than they'd like to on their own. All of our kids have needed to learn to focus their attention, and three is a good time to make it a habit.
We spend a lot of time outside when the weather permits, and I find the little ones are happier that way. We are blessed to be on a farm, so this is not difficult. But even when we lived briefly in town, we were outside a lot too. A small garden to look after is really nice for a little guy (though you'll do most of the real work :) Ours just planted some flower seeds the other day, and is faithfully watering (or drowning) them in anticipation of watching them grow. If you don't have room for a garden, even a few pots of plants are nice to look after.
Some things we have inside that are fun:
A sandbox with a cover (I know this sounds like an outdoor thing, and it is during nice months, but we have Canadian winters so must do what we can). I keep it downstairs and there are pretty strict rules for keeping the sand inside the box (within reason).
A Runners bike which is a bike without pedals. We allow it indoors (but only in winter), but the setup of our house allows for that. Any kind of ride on toy is nice, I like this one because it is quiet (it has rubber wheels).
A few puzzles, some easier to be done mostly on their own, some more difficult to be done together. Our almost turned three year old likes to try and match the pieces by picture, but he still needs help.
Play Dough or Moon Dough or clay for creating.
A magnifying glass to study things - rocks, cheerios, bugs, shells, etc.
A box of different kinds of beads to sort, or put on a string.
Bingo dabbers for "painting".
A tub of water with a boat, whale, octopus, whatever else is fun. This can be done in the bathtub, but seems to be more interesting in the kitchen sink or in a rubbermaid on the floor.
I personally think it is never too young to break up activities the same way you would school subjects - after bike riding/going walking/some kind of physical activity, do something with thinking, like a puzzle or sorting beads, than read a story afterward.
Everyone likes to have some stories throughout the day. I try to set aside a couple of times to read with them and do a few short activities, then they are more content to go and play for a while.
Don't forget chores! A three year old can be useful, and is happier if he is. Making beds, picking up toys, feeding a pet, watering plants, vacuuming or sweeping are all things that a little guy can do (even if you have to do it over again later).
Last but not least, it is my personal opinion that kids are better with little to no TV or electronics. Ours are much more content and able to focus without these things. Again, this is just my opinion.
I am sure many others will have good advice for you.