@Ren couple things:
First, the idea of no bottom level does not at all require that the levels themselves every be anything other than physical: to capture what I'm going for, the analogy wouldn't be physical vs mathematical, but rather Newtonian physics vs quantum mechanics or chemistry vs physics (that is, more fundamental levels within physical science itself). The idea in physics is to hope there is some bottom level, say string theory, that tells you what the most fundamental constituents are -- however, there's a worry if you can keep splitting into finer and finer parts, so that, even if all of them are physicalist in nature, there's no final level like string theory ... there might be a string-2 theory that is more fundamental, and so on.
In fact, if the world turned out to be entirely physical and with a finite number of levels, it is child's play for me to cook up a world with further levels -- I really just need to write down another mathematical structure in which the physical world's 'final level' is embedded.
Part of the reason this is child's play is how thin a metaphysics physics gives -- where it's very closely determined by the mathematical structure.
Now, as to the question of whether there could be something below the physical, right now there's very good reason to worry that, because there seems to be virtually no other way to propose how to integrate the mental and physical (without severely deflating one or the other, which I think so far nobody has shown a plausible account of how to do) and since causal-closure considerations give strong motivation for avoiding dualism.
I think there are lots of accounts of more specific answers in the neutral monism or panpsychism type spirit (I favor the first so far), some version of which are not easily ruled out developed by philosophers as to what that more fundamental level would be, though as a factual matter, we're not in a position to know if any is the right account of our world yet.
So I do think sadly the reality really is that we might be more confident there's something further to be discovered than we are as to what the answer's form will be exactly.
(Some are pessimistic and actually argue we won't be able to find out, though I don't go that far yet.)
As a note, I don't have any reason to positively suspect neutral monism will involve infinite layers -- it was just a theoretical consideration to illustrate a specific way in which there may be no fundamental layer to metaphysics.... rather than something I think necessarily true of our world.
I don't conjecture what the final story is less because I think it's hard to imagine anything that could work and more because I just think they're all somewhat arbitrary shots in the dark at this current stage.
Of course I love playing with possible theories, but at this point I think they'd just be ways *A WORLD* with our mathematical structure might be; not speculations as to how
our specific world plausibly might turn out -- I think what we're lacking is specific facts about consciousness or physics in our world that would help us decide among the competing monist theories.