An update from more "testing"; the low-quality models/scenery pop-in issue came back when I started adding more vehicles to my popgroup (Im sitting at some 250+ added vehicles now iirc). I tried bumping the physical/virtual memory values higher, but to no avail.
While testing different traffic density values to get more parked cars to spawn around the map, I found out that upping the ped- and vehicle- memory multipliers in <configpopulation> in the gameconfig apparently helps control those memory issues (I think).
I set them to 5000 each (again, probably overkill) and now I have almost no "low quality" model issues, or scenery pop-in when looking in different directions quickly.
So what does the physical/virtual memory control? Past a certain point, upping these values seems to give diminishing returns (if any to begin with), atleast for my game.
I have left them back at 1500/5000 for now since it doesnt seem to be harming my game, but I probably could lower them (or perhaps reset them to default values). Maybe someone with a more "intensive" mod folder (4K textures for everything, real car models, etc.) should test those if they have issues with texture/model pop-ins.