Mount a 1/4"-diameter socket to the side of your workbench to make changing bits quicker.
I purchased a mini chuck that features a 1/4"-diameter hex shank to hold small drill bits securely. The instructions suggest clamping the shank of the chuck in a vise while you hand-tighten the bits in the chuck. To speed up bit changes, I drilled a hole large enough to accommodate a 1/4"-diameter socket in my workbench near the drill press and used cyanoacrylate (CA) glue to lock the socket into the hole. Now, I slide the shank of the chuck into the socket to change bits quickly.