It's a bit of a mystery.
In most of the languages of Europe the word for dog comes from an ancient PIE root which was something like kwon- and which appears in e.g. Latin canis, Ancient Greek kyon and Welsh corgi. In Germanic languages the k sound became h, so we get Modern German Hund and Old English hund (another example of this consonant shift is Greek kardia versus English heart, German Hertz).
For today 'hound' is reserved for dogs (or people) that hunt or track something bloodhound, newshound, or breeds of hunting dogs wolfhound, or used metaphorically, or perhaps in a jokey way. At some point in late Old English a word docga (possibly referring to a spacial breed of strong or powerful animal) emerged from completely unknown origins. During the Middle English period dog displaced hound as the standard word for members of the genus Canis and the meaning of hound narrowed to the more restricted sense it is used in today.
To me hound has more portentous feel than dog: "The Dog of the Baskervilles" wouldn't really worked.