I have a theory that what people find most interesting in music is self-reference, recursion and fractal-like scale-invariance. Rhythms based on powers of two are a great way to get this kind of recursion because they can be compounded or subdivided so easily. A bar of four can be treated as two bars of two, or half of a bar of eight. You can further subdivide your bars into quarter, eighth and sixteenth notes. You can group your bars of four or eight into four or eight or sixteen-bar phrases. Here’s a visual representation of this kind of powers-of-two recursion:
To keep all the symmetry from getting oppressive, you can throw a few multiples of three or five in there, for example with triplet figures or ten-bar phrases, knowing that listeners can easily parse these against the powers-of-two baseline. Even in 3/4 time, you usually subdivide beats and group phrases by powers of two.
As Laszlo B. Tamas says, there’s a limit to the number of discrete pieces of information we can keep in our working memory at any one time. I don’t know what it’s like to be a Bulgarian or Indian musician, but like Laszlo, I can only conceive of the more complex time signatures in terms of groups of simpler ones, like thinking of 11/8 as 3+3+3+2. Four is a big enough number to be subdivided interestingly, but still small enough so as to not be overwhelming.