At Parents.com in an article, Making Math Fun, "little kids naturally love counting, sorting, doing puzzles, and discovering patterns. But once these activities get labeled "math," with daily doses of addition, multiplication, fractions, and long division, many children lose both confidence and interest." See the entire article at Making Math Fun
They suggest activities to peak their attention and fun meter. I personnally use cooking and baking activities to bring math into our daily routine. Using everyday situations can really help without making it seem like "math". In addition, there's no prep time required! For example, you can always ask you toddler to help in the grocery store, by letting him pick and count 5 red apples or ask questions like, How many different color apples do you see?
If you find yourself frustrated and not sure what to do next...here's an article that provides 4 keys to Making Math Fun--http://hubpages.com/hub/Make-Math-FUN. Its a great resource for parents or teachers that can save you time and frustration!
Click it out--Its called Make Math More Fun!
There are alot of resources available to you. Here's a video that may trigger activities for sorting! Remember, of most importance is you and your child having FUN together! Enjoy!
The video is less than 2 minutes long and the "game" requires very little prep.