How do you know the address shown on your hardware wallet belongs to you?

I've been thinking of setting up a multisig wallet using my Ledger Nano S and Trezor One so I was reading about it online and came across this article. In it the writer says that using a Ledger for multisig is less secure than singlesig since Ledger doesn't show the multisig's xpubs on the device. He says that if you only use the xpub shown on your computer screen you have no way of knowing if it belongs to your device or an attacker's device.

This made me ask the question: How do you even know that addresses shown on hardware wallets actually belong to you? If your computer is infected with malware what is stopping that malware from feeding a fake address belonging to the attacker to your hardware wallet?