Here is a excerpt from The Linux Kernel guide produced by The Linux Documentation Project (TLDP).
Modules can be removed using the rmmod command but demand loaded modules are automatically removed from the system by kerneld when they are no longer being used. Every time its idle timer expires, kerneld makes a system call requesting that all unused demand loaded modules are removed from the system. The timer's value is set when you start kerneld; my kerneld checks every 180 seconds. So, for example, if you mount an iso9660 CD ROM and your iso9660 filesystem is a loadable module, then shortly after the CD ROM is unmounted, the iso9660 module will be removed from the kernel.
Getting rid of unused modules from memory is a good idea but I didn't find a way to doing this with latest 2.6.x kernels. Can anyone throw some light on this?