Reviewing some docs further, I guess I could give the RAM its own AXI4 Slave Interface by creating a bus object (or multiple ports) with the right signals. It would add more lines to the block design but I guess Vivado would be OK with that.
On the other hand, maybe not. Saw other documentation saying you can have either an AXI4-Lite Slave or a AXI4 Slave but not both. You can have multiple stream masters or slaves, but not multiple register interfaces.