Hi guys, been a month. I'm writing this because I don't want to debug a panic involving the amdkfd driver... but it does now link and load!
So AMD recently introduced a new therock-pr-bot which, um, I'm not the biggest fan of. It doesn't seem to really be thinking that far ahead of its nose. Previously I have submitted a PR to ROCm/rocm-systems and got it accepted; it was a small pull request to test the waters. It got a bit abandoned, I pinged again, and it got instantly merged.
Then I think about ~2 weeks ago I decided to start submitting more PRs ( 1 , and 2 ). Following the first one, I got hit with "PR Check — Action Required" and "Not Ready To Review", and I couldn't get it to pass lol. I tried adding the tests (was then told to remove them lmao), then I changed the title, but yeah. It was slightly cooked. Then I submitted the second one; this time I decided to bunch a lot of the header changes into one cause doing it file by file was going to be painful. It got reviewed pretty quick—we had a quick discussion on replacing a syscall(__NR_memfd_create, ring_buf_shm_path, 0); with just a straight syscall. The thinking was that the min required versions of libraries on the platforms which ROCm is supported on all have memfd_create, so why are we trying to also keep a legacy syscall? We decided to completely remove it in the end.
Beyond that one technical issue, it was all just normal-level issues—i.e. I made a mistake in terms of preprocessor macros. Otherwise it's been fine. They're still not merged in, but I assume it'll be merged soon. Next two PRs are on the build system, then finally adding in the FreeBSD changes :D
No build system is good, all are terrible, all need to be destroyed (same applies to package managers), none of them are fun to use, and they always cause me sadness and despair. drm-kmod's build system is no different, albeit it was not painful to use.
To begin, to get ROCm working, I need to get amdkfd working. Now amdkfd basically binds itself to the amdgpu code, which for the most part already works! The Makefile at the root of drm-kmod somehow calls the Makefile within the amd/ directory, which then calls the Makefile within amdgpu which then actually builds amdgpu. Sounds good and all—there is a folder for amdkfd, so I had a quick glance in, saw the legacy Makefiles, tried to run them... and hit a million errors cause files were missing. So instead I did the next best thing: I just used the amdgpu Makefile.
I literally just checked out Linux kernel v6.12, copied over the amdkfd driver files to the drivers/drm/gpu/amd/amdkfd folder, added that to the SRCDIR stuff, then added the needed defines, and we were off to the races! If the races were fighting undefined errors, redefinitions, and type errors...? I wasn't expecting the last one.
On a quick random side tangent: I don't know why, but adding stuff line by line using the \ thingy at the end of the line to go to the next line didn't work for some reason, so I just had one massive line:
+ kfd_module.c kfd_device.c kfd_chardev.c kfd_process.c kfd_pasid.c \
kfd_topology.c kfd_doorbell.c kfd_mqd_manager.c kfd_device_queue_manager.c \
kfd_kernel_queue.c kfd_interrupt.c kfd_mqd_manager_v11.c \
amdgpu_amdkfd_gpuvm.c kfd_events.c kfd_queue.c kfd_flat_memory.c \
kfd_process_queue_manager.c kfd_freebsd_stubs.c \
kfd_device_queue_manager_v11.c kfd_int_process_v11.c kfd_packet_manager.c So there's this thing called a kthread in the Linux kernel which is literally just a background kernel process that does work in the background (think flushing to disk or smth). FreeBSD's equivalent to a Linux kthread is a combination of a kernel thread and a kernel process. They're both different and neither individually fulfills the requirements of a Linux kthread, but combined they are equivalent to a Linux kthread.
The amdkfd driver uses kthread_use_mm (and kthread_unuse_mm) quite a bit. Whenever they get used, I do need to do a little bit of manual editing—mainly adding in two namespaces and swapping out the calls with different ones. But that's pretty simple, just took a while to figure out. Probably the most annoying one to fix was the read_user_wptr macro, which required some creative engineering to get it to work :/
For read_user_wptr, I ended up hacking it with FreeBSD's vmspace_switch_aio to manually flip the thread's VM space before reading the user pointer with get_user. I think this is ok, but I'm not fully sure. It compiles though! (macro is below)
#if defined(__linux__)
#define read_user_wptr(mmptr, wptr, dst) \
({ \
bool valid = false; \
if ((mmptr) && (wptr)) { \
pagefault_disable(); \
if ((mmptr) == current->mm) { \
valid = !get_user((dst), (wptr)); \
} else if (current->flags & PF_KTHREAD) { \
kthread_use_mm(mmptr); \
valid = !get_user((dst), (wptr)); \
kthread_unuse_mm(mmptr); \
} \
pagefault_enable(); \
} \
valid; \
})
#elif defined(__FreeBSD__)
#include
#include
#define read_user_wptr(mmptr, wptr, dst) \
({ \
bool valid = false; \
if ((mmptr) && (wptr)) { \
pagefault_disable(); \
if ((mmptr) == current->mm) { \
valid = !get_user((dst), (wptr)); \
} else if (curthread->td_pflags & TDP_KTHREAD) { \
struct vmspace *target_vm = \
(struct vmspace *)(void *)(mmptr); \
struct vmspace *orig_vm = \
curthread->td_proc->p_vmspace; \
\
vmspace_switch_aio(target_vm); \
valid = !get_user((dst), (wptr)); \
vmspace_switch_aio(orig_vm); \
} \
pagefault_enable(); \
} \
valid; \
})
#endif
While getting it working I had to port over a bunch of platform-specific stuff. I decided to just port over the ones needed for my laptop (Radeon 780M), which was just v11. If you'd like to port over yours, copy over the required files from the Linux kernel source, modify them as needed, and then remove my stubs for the device queue and event handlers.
I stubbed out all the debugfs stuff because I cannot be bothered to try and fix it tbh—it is not worth my time to try and get it working ;-;
So pr_debug_ratelimited shows up from time to time within the driver; I just set a macro which replaces it with pr_debug. I found out about this macro during a merge request in Phabricator lol.
Under the reuse_dmamap stuff, I literally just return false. I'm not even going to try to support multi-GPU setups at this stage; it'd be very hard.
To be frank, a lot of this is just swapping out random functions and stuff—none of it is groundbreaking. Sometimes some of the DMA/memory stuff would get a little bit screwy, but that was it.
I haven't actually spent much time looking into why the gun explodes. But I have had quick glances at the core dump. It seems to be centered around trying to make /dev/kfd twice, and during the second time it gets an "already exists" code, which in FreeBSD results in a panic unless you pass a certain value (which is not passed by LinuxKPI). So I'm gonna go do more reading into that after I finish this post ;-;.
super-special branch, and develop off of there. Current commit is a7b4e762461510b11b3881e0238eb462d37712b4 . Yeah! Support and testing on other platforms is kinda needed right now.
There are some which are still pending reviews: https://reviews.freebsd.org/D58207, https://reviews.freebsd.org/D58228, https://reviews.freebsd.org/D58222