This practice used to be about hygiene. Hats were worn to cover your head while you worked outside. As result hats were covered with dirt and sweat.

So it was expected that it was to be taken off.

People who didn’t take their hat off were considered unclean and rude to the host.

In India it is a sign of respect to the family or business you are entering.