There are two types of drivers:
Internal drivers that are part of the kernel binary and are therefore part of kernel memory when the system is running, and
external drivers that are not part of a kernel binary which are loaded into user memory when required.
These external drivers are also known as loadable kernel modules.
Loadable modules exist as disk files, stored under /lib/modules. These can be loaded into user memory as required. They can also be unloaded from user memory when you don't need them anymore.