Core: Pooled Memory Fixes (#2895)

* Update sceKernelMemoryPoolExpand

Hardware tests show that this function is basically the same as sceKernelAllocateDirectMemory, with some minor differences.
Update the memory searching code to match my updated AllocateDirectMemory code, with appropriate error conditions.

* Update MemoryPoolReserve

Only difference between real hw and our code is behavior with addr = 0.

* Don't coalesce PoolReserved areas.

Real hardware doesn't coalesce them.

* Update PoolCommit

Plenty of edge case behaviors to handle here.
Addresses are treated as fixed, EINVAL is returned for bad mappings, name should be preserved from PoolReserving, committed areas should coalesce, reserved areas get their phys_base updated

* Formatting

* Adjust fixed PoolReserve path

Hardware tests suggest this will overwrite all VMAs in the range. Run UnmapMemoryImpl on the full area, then reserve. Same logic applies to normal reservations too.

Also adjusts logic of the non-fixed path to more closely align with hardware observations.

* Remove phys_base modifications

This can be handled later. Doing the logic properly would likely take work in MergeAdjacent, and would probably need to be applied to normal dmem mappings too.

* Use VMAHandle.Contains()

Why do extra math when we have a function specifically for this?

* Update memory.cpp

* Remove unnecessary code

Since I've removed those two asserts, these two lines of code effectively do nothing.

* Clang

* Fix names

* Fix PoolDecommit

Should fix the address space regressions in UE titles on Windows.

* Fix error log

Should make the cause of this clearer?

* Clang

* Oops

* Remove coalesce on PoolCommit

Windows makes this more difficult.

* Track pool budgets

If you try to commit more pooled memory than is allocated, PoolCommit returns ENOMEM.
Also fixes error conditions for PoolDecommit, that should return EINVAL if given an address that isn't part of the pool.

Note: Seems like the pool budget can't hit zero? I used a <= comparison based on hardware tests, otherwise we're able to make more mappings than real hardware can.

src/core/memory.h

@@ -198,7 +198,7 @@ public:
     int MapFile(void** out_addr, VAddr virtual_addr, size_t size, MemoryProt prot,
                 MemoryMapFlags flags, uintptr_t fd, size_t offset);
 
-    void PoolDecommit(VAddr virtual_addr, size_t size);
+    s32 PoolDecommit(VAddr virtual_addr, size_t size);
 
     s32 UnmapMemory(VAddr virtual_addr, size_t size);
 
@@ -274,6 +274,7 @@ private:
     size_t total_direct_size{};
     size_t total_flexible_size{};
     size_t flexible_usage{};
+    size_t pool_budget{};
     Vulkan::Rasterizer* rasterizer{};
 
     friend class ::Core::Devtools::Widget::MemoryMapViewer;