If someone "borrows" your thumb drive they could extract the secrets and return it without you knowing, but AFAIK secrets cannot be extracted from a real token such as a Yubikey.

It doesn't just work by reading a passkey.

From the webpage:

* Non-intrusive. pam_usb doesn’t require any modifications of the USB storage device to work (no additional partitions required).

* USB Serial number, model and vendor verification.

So is it only for local console authentication? There's no way to read that stuff remotely.

Come on mate, just use a bit of imagination. You use the pam_usb module to login locally and an encrypted passkey stored on the same usb thumbdrive for remote SSH connections.