I mean, having config files in there makes a lot of sense. It allows you to move your home to some other machine/install and immediately feel at home, because everything is configured to your liking.

Now, the fact that many apps create their own directory instead of putting their stuff in ~/.config is pretty annoying.