Are you storing sensitive data on your computer. Is there more than one user for your computer ? If answer to both these question is yes, then you must be worried about your data security as well. USB Drive is easiest way to carry data from one computer to another. For multi user PC where you cannot disable USB, you can at least monitor USB connection with utilities like USBLogView.
USBLogView is one free utility which monitors all USB connection, and device connected to a computer once installed and running. If you suspect any activity you can leave it running and it will prepare a log of each USB drive connected to your computer. It only monitors plug-in and unplug activity though while it is running.