OS X has an alternate, and I think better, menu format for dock folders. With the alternate menus you have bigger icons, a cleaner look, better scrolling for long lists, and you can click into nested folders right in the menu.
To enable the alternate view, open Terminal and type:
defaults write com.apple.dock use-new-list-stack -boolean YES; killall Dock
If you find you don’t like these changes, type the command again, replacing YES with NO.