video_core: Implement an arm64 shader-jit backend (#7002)
* externals: Add oaksim submodule Used for emitting ARM64 assembly * common: Implement aarch64 ABI Utilize oaknut to implement a stack frame. * tests: Allow shader-jit tests for x64 and a64 Run the shader-jit tests for both x86_64 and arm64 targets * video_core: Initialize arm64 shader-jit backend Passes all current unit tests! * shader_jit_a64: protect/unprotect memory when jit-ing Required on MacOS. Memory needs to be fully unprotected and then re-protected when writing or there will be memory access errors on MacOS. * shader_jit_a64: Fix ARM64-Imm overflow These conditionals were throwing exceptions since the immediate values were overflowing the available space in the `EOR` instructions. Instead they are generated from `MOV` and then `EOR`-ed after. * shader_jit_a64: Fix Geometry shader conditional * shader_jit_a64: Replace `ADRL` with `MOVP2R` Fixes some immediate-generation exceptions. * common/aarch64: Fix CallFarFunction * shader_jit_a64: Optimize `SantitizedMul` Co-authored-by: merryhime <merryhime@users.noreply.github.com> * shader_jit_a64: Fix address register offset behavior Based on https://github.com/citra-emu/citra/pull/6942 Passes unit tests. * shader_jit_a64: Fix `RET` address offset A64 stack is 16-byte aligned rather than 8. So a direct port of the x64 code won't work. Fixes weird branches into invalid memory for any shaders with subroutines. * shader_jit_a64: Increase max program size Tuned for A64 program size. * shader_jit_a64: Use `UBFX` for extracting loop-state Co-authored-by: JosJuice <JosJuice@users.noreply.github.com> * shader_jit_a64: Optimize `SUB+CMP` to `SUBS` Co-authored-by: JosJuice <JosJuice@users.noreply.github.com> * shader_jit_a64: Optimize `CMP+B` to `CBNZ` Co-authored-by: JosJuice <JosJuice@users.noreply.github.com> * shader_jit_a64: Use `FMOV` for `ONE` vector Co-authored-by: JosJuice <JosJuice@users.noreply.github.com> * shader_jit_a64: Remove x86-specific documentation * shader_jit_a64: Use `UBFX` to extract exponent Co-authored-by: JosJuice <JosJuice@users.noreply.github.com> * shader_jit_a64: Remove redundant MIN/MAX `SRC2`-NaN check Special handling only needs to check SRC1 for NaN, not SRC2. It would work as follows in the four possible cases: No NaN: No special handling needed. Only SRC1 is NaN: The special handling is triggered because SRC1 is NaN, and SRC2 is picked. Only SRC2 is NaN: FMAX automatically picks SRC2 because it always picks the NaN if there is one. Both SRC1 and SRC2 are NaN: The special handling is triggered because SRC1 is NaN, and SRC2 is picked. Co-authored-by: JosJuice <JosJuice@users.noreply.github.com> * shader_jit/tests:: Add catch-stringifier for vec2f/vec3f * shader_jit/tests: Add Dest Mask unit test * shader_jit_a64: Fix Dest-Mask `BSL` operand order Passes the dest-mask unit tests now. * shader_jit_a64: Use `MOVI` for DestEnable mask Accelerate certain cases of masking with MOVI as well Co-authored-by: JosJuice <JosJuice@users.noreply.github.com> * shader_jit/tests: Add source-swizzle unit test This is not expansive. Generating all `4^4` cases seems to make Catch2 crash. So I've added some component-masking(non-reordering) tests based on the Dest-Mask unit-test and some additional ones to test broadcasts/splats and component re-ordering. * shader_jit_a64: Fix swizzle index generation This was still generating `SHUFPS` indices and not the ones that we wanted for the `TBL` instruction. Passes all unit tests now. * shader_jit/tests: Add `ShaderSetup` constructor to `ShaderTest` Rather than using the direct output of `CompileShaderSetup` allow a `ShaderSetup` object to be passed in directly. This enabled the ability emit assembly that is not directly supported by nihstro. * shader_jit/tests: Add `CALL` unit-test Tests nested `CALL` instructions to eventually reach an `EX2` instruction. EX2 is picked in particular since it is implemented as an even deeper dispatch and ensures subroutines are properly implemented between `CALL` instructions and implementation-calls. * shader_jit_a64: Fix nested `BL` subroutines `lr` was getting writen over by nested calls to `BL`, causing undefined behavior with mixtures of `CALL`, `EX2`, and `LG2` instructions. Each usage of `BL` is now protected with a stach push/pop to preserve and restore teh `lr` register to allow nested subroutines to work properly. * shader_jit/tests: Allocate generated tests on heap Each of these generated shader-test objects were causing the stack to overflow. Allocate each of the generated tests on the heap and use unique_ptr so they only exist within the life-time of the `REQUIRE` statement. * shader_jit_a64: Preserve `lr` register from external function calls `EMIT` makes an external function call, and should be preserving `lr` * shader_jit/tests: Add `MAD` unit-test The Inline Asm version requires an upstream fix: https://github.com/neobrain/nihstro/issues/68 Instead, the program code is manually configured and added. * shader_jit/tests: Fix uninitialized instructions These `union`-type instruction-types were uninitialized, causing tests to indeterminantly fail at times. * shader_jit_a64: Remove unneeded `MOV` Residue from the direct-port of x64 code. * shader_jit_a64: Use `std::array` for `instr_table` Add some type-safety and const-correctness around this type as well. * shader_jit_a64: Avoid c-style offset casting Add some more const-correctness to this function as well. * video_core: Add arch preprocessor comments * common/aarch64: Use X16 as the veneer register https://developer.arm.com/documentation/102374/0101/Procedure-Call-Standard * shader_jit/tests: Add uniform reading unit-test Particularly to ensure that addresses are being properly truncated * common/aarch64: Use `X0` as `ABI_RETURN` `X8` is used as the indirect return result value in the case that the result is bigger than 128-bits. Principally `X0` is the general-case return register though. * common/aarch64: Add veneer register note `LR` is generally overwritten by `BLR` anyways, and would also be a safe veneer to utilize for far-calls. * shader_jit_a64: Remove unneeded scratch register from `SanitizedMul` * shader_jit_a64: Fix CALLU condition Should be `EQ` not `NE`. Fixes the regression on Kid Icarus. No known regressions anymore! --------- Co-authored-by: merryhime <merryhime@users.noreply.github.com> Co-authored-by: JosJuice <JosJuice@users.noreply.github.com>
This commit is contained in:
Wunk
GitHub
parent
3218af38d0
commit
e13735b624
14 changed files with 1874 additions and 25 deletions
|
|
@ -53,6 +53,8 @@ add_custom_command(OUTPUT scm_rev.cpp
|
|||
add_library(citra_common STATIC
|
||||
aarch64/cpu_detect.cpp
|
||||
aarch64/cpu_detect.h
|
||||
aarch64/oaknut_abi.h
|
||||
aarch64/oaknut_util.h
|
||||
alignment.h
|
||||
android_storage.h
|
||||
android_storage.cpp
|
||||
|
|
@ -179,6 +181,10 @@ if ("x86_64" IN_LIST ARCHITECTURE)
|
|||
target_link_libraries(citra_common PRIVATE xbyak)
|
||||
endif()
|
||||
|
||||
if ("arm64" IN_LIST ARCHITECTURE)
|
||||
target_link_libraries(citra_common PRIVATE oaknut)
|
||||
endif()
|
||||
|
||||
if (CITRA_USE_PRECOMPILED_HEADERS)
|
||||
target_precompile_headers(citra_common PRIVATE precompiled_headers.h)
|
||||
endif()
|
||||
|
|
|
|||
155
src/common/aarch64/oaknut_abi.h
Normal file
155
src/common/aarch64/oaknut_abi.h
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
// Copyright 2023 Citra Emulator Project
|
||||
// Licensed under GPLv2 or any later version
|
||||
// Refer to the license.txt file included.
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "common/arch.h"
|
||||
#if CITRA_ARCH(arm64)
|
||||
|
||||
#include <bitset>
|
||||
#include <initializer_list>
|
||||
#include <oaknut/oaknut.hpp>
|
||||
#include "common/assert.h"
|
||||
|
||||
namespace Common::A64 {
|
||||
|
||||
constexpr std::size_t RegToIndex(const oaknut::Reg& reg) {
|
||||
ASSERT(reg.index() != 31); // ZR not allowed
|
||||
return reg.index() + (reg.is_vector() ? 32 : 0);
|
||||
}
|
||||
|
||||
constexpr oaknut::XReg IndexToXReg(std::size_t reg_index) {
|
||||
ASSERT(reg_index <= 30);
|
||||
return oaknut::XReg(static_cast<int>(reg_index));
|
||||
}
|
||||
|
||||
constexpr oaknut::VReg IndexToVReg(std::size_t reg_index) {
|
||||
ASSERT(reg_index >= 32 && reg_index < 64);
|
||||
return oaknut::QReg(static_cast<int>(reg_index - 32));
|
||||
}
|
||||
|
||||
constexpr oaknut::Reg IndexToReg(std::size_t reg_index) {
|
||||
if (reg_index < 32) {
|
||||
return IndexToXReg(reg_index);
|
||||
} else {
|
||||
return IndexToVReg(reg_index);
|
||||
}
|
||||
}
|
||||
|
||||
inline constexpr std::bitset<64> BuildRegSet(std::initializer_list<oaknut::Reg> regs) {
|
||||
std::bitset<64> bits;
|
||||
for (const oaknut::Reg& reg : regs) {
|
||||
bits.set(RegToIndex(reg));
|
||||
}
|
||||
return bits;
|
||||
}
|
||||
|
||||
constexpr inline std::bitset<64> ABI_ALL_GPRS(0x00000000'7FFFFFFF);
|
||||
constexpr inline std::bitset<64> ABI_ALL_FPRS(0xFFFFFFFF'00000000);
|
||||
|
||||
constexpr inline oaknut::XReg ABI_RETURN = oaknut::util::X0;
|
||||
constexpr inline oaknut::XReg ABI_PARAM1 = oaknut::util::X0;
|
||||
constexpr inline oaknut::XReg ABI_PARAM2 = oaknut::util::X1;
|
||||
constexpr inline oaknut::XReg ABI_PARAM3 = oaknut::util::X2;
|
||||
constexpr inline oaknut::XReg ABI_PARAM4 = oaknut::util::X3;
|
||||
|
||||
constexpr std::bitset<64> ABI_ALL_CALLER_SAVED = 0xffffffff'4000ffff;
|
||||
constexpr std::bitset<64> ABI_ALL_CALLEE_SAVED = 0x0000ff00'7ff80000;
|
||||
|
||||
struct ABIFrameInfo {
|
||||
u32 subtraction;
|
||||
u32 fprs_offset;
|
||||
};
|
||||
|
||||
inline ABIFrameInfo ABI_CalculateFrameSize(std::bitset<64> regs, std::size_t frame_size) {
|
||||
const size_t gprs_count = (regs & ABI_ALL_GPRS).count();
|
||||
const size_t fprs_count = (regs & ABI_ALL_FPRS).count();
|
||||
|
||||
const size_t gprs_size = (gprs_count + 1) / 2 * 16;
|
||||
const size_t fprs_size = fprs_count * 16;
|
||||
|
||||
size_t total_size = 0;
|
||||
total_size += gprs_size;
|
||||
const size_t fprs_base_subtraction = total_size;
|
||||
total_size += fprs_size;
|
||||
total_size += frame_size;
|
||||
|
||||
return ABIFrameInfo{static_cast<u32>(total_size), static_cast<u32>(fprs_base_subtraction)};
|
||||
}
|
||||
|
||||
inline void ABI_PushRegisters(oaknut::CodeGenerator& code, std::bitset<64> regs,
|
||||
std::size_t frame_size = 0) {
|
||||
using namespace oaknut;
|
||||
using namespace oaknut::util;
|
||||
auto frame_info = ABI_CalculateFrameSize(regs, frame_size);
|
||||
|
||||
// Allocate stack-space
|
||||
if (frame_info.subtraction != 0) {
|
||||
code.SUB(SP, SP, frame_info.subtraction);
|
||||
}
|
||||
|
||||
// TODO(wunk): Push pairs of registers at a time with STP
|
||||
std::size_t offset = 0;
|
||||
for (std::size_t i = 0; i < 32; ++i) {
|
||||
if (regs[i] && ABI_ALL_GPRS[i]) {
|
||||
const XReg reg = IndexToXReg(i);
|
||||
code.STR(reg, SP, offset);
|
||||
offset += 8;
|
||||
}
|
||||
}
|
||||
|
||||
offset = 0;
|
||||
for (std::size_t i = 32; i < 64; ++i) {
|
||||
if (regs[i] && ABI_ALL_FPRS[i]) {
|
||||
const VReg reg = IndexToVReg(i);
|
||||
code.STR(reg.toQ(), SP, u16(frame_info.fprs_offset + offset));
|
||||
offset += 16;
|
||||
}
|
||||
}
|
||||
|
||||
// Allocate frame-space
|
||||
if (frame_size != 0) {
|
||||
code.SUB(SP, SP, frame_size);
|
||||
}
|
||||
}
|
||||
|
||||
inline void ABI_PopRegisters(oaknut::CodeGenerator& code, std::bitset<64> regs,
|
||||
std::size_t frame_size = 0) {
|
||||
using namespace oaknut;
|
||||
using namespace oaknut::util;
|
||||
auto frame_info = ABI_CalculateFrameSize(regs, frame_size);
|
||||
|
||||
// Free frame-space
|
||||
if (frame_size != 0) {
|
||||
code.ADD(SP, SP, frame_size);
|
||||
}
|
||||
|
||||
// TODO(wunk): Pop pairs of registers at a time with LDP
|
||||
std::size_t offset = 0;
|
||||
for (std::size_t i = 0; i < 32; ++i) {
|
||||
if (regs[i] && ABI_ALL_GPRS[i]) {
|
||||
const XReg reg = IndexToXReg(i);
|
||||
code.LDR(reg, SP, offset);
|
||||
offset += 8;
|
||||
}
|
||||
}
|
||||
|
||||
offset = 0;
|
||||
for (std::size_t i = 32; i < 64; ++i) {
|
||||
if (regs[i] && ABI_ALL_FPRS[i]) {
|
||||
const VReg reg = IndexToVReg(i);
|
||||
code.LDR(reg.toQ(), SP, frame_info.fprs_offset + offset);
|
||||
offset += 16;
|
||||
}
|
||||
}
|
||||
|
||||
// Free stack-space
|
||||
if (frame_info.subtraction != 0) {
|
||||
code.ADD(SP, SP, frame_info.subtraction);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace Common::A64
|
||||
|
||||
#endif // CITRA_ARCH(arm64)
|
||||
43
src/common/aarch64/oaknut_util.h
Normal file
43
src/common/aarch64/oaknut_util.h
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
// Copyright 2023 Citra Emulator Project
|
||||
// Licensed under GPLv2 or any later version
|
||||
// Refer to the license.txt file included.
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "common/arch.h"
|
||||
#if CITRA_ARCH(arm64)
|
||||
|
||||
#include <type_traits>
|
||||
#include <oaknut/oaknut.hpp>
|
||||
#include "common/aarch64/oaknut_abi.h"
|
||||
|
||||
namespace Common::A64 {
|
||||
|
||||
// BL can only reach targets within +-128MiB(24 bits)
|
||||
inline bool IsWithin128M(uintptr_t ref, uintptr_t target) {
|
||||
const u64 distance = target - (ref + 4);
|
||||
return !(distance >= 0x800'0000ULL && distance <= ~0x800'0000ULL);
|
||||
}
|
||||
|
||||
inline bool IsWithin128M(const oaknut::CodeGenerator& code, uintptr_t target) {
|
||||
return IsWithin128M(code.ptr<uintptr_t>(), target);
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
inline void CallFarFunction(oaknut::CodeGenerator& code, const T f) {
|
||||
static_assert(std::is_pointer_v<T>, "Argument must be a (function) pointer.");
|
||||
const std::uintptr_t addr = reinterpret_cast<std::uintptr_t>(f);
|
||||
if (IsWithin128M(code, addr)) {
|
||||
code.BL(reinterpret_cast<const void*>(f));
|
||||
} else {
|
||||
// X16(IP0) and X17(IP1) is the standard veneer register
|
||||
// LR is also available as an intermediate register
|
||||
// https://developer.arm.com/documentation/102374/0101/Procedure-Call-Standard
|
||||
code.MOVP2R(oaknut::util::X16, reinterpret_cast<const void*>(f));
|
||||
code.BLR(oaknut::util::X16);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace Common::A64
|
||||
|
||||
#endif // CITRA_ARCH(arm64)
|
||||
Loading…
Add table
Add a link
Reference in a new issue