Bleeding on the Edge: app-emulation/virtualbox-modules-5.1.22 vs. kernel 4.12.0-rc2

Note that kernel 4.12.0 has been released and requires additional steps outlined in a new post:
app-emulation/virtualbox-modules-5.1.22 vs. kernel 4.12.0 final

If you’re running the latest kernel from the mainline branch like I am, you may have noticed that nvidia drivers compile with no problem, for once. This is a rare occurrence and always reason to celebrate. However, Oracle’s VirtualBox modules rear their ugly heads again. If you use VirtualBox and have recently run updates, you probably saw output something like this:

In file included from /var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/alloc-r0drv-linux.c:31:0:
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/alloc-r0drv-linux.c: In function ‘VBoxHost_RTMemContAlloc’:
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/the-linux-kernel.h:323:47: error: implicit declaration of function ‘set_pages_x’ [-Werror=implicit-function-declaration]
 # define MY_SET_PAGES_EXEC(pPages, cPages)    set_pages_x(pPages, cPages)
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/alloc-r0drv-linux.c:445:13: note: in expansion of macro ‘MY_SET_PAGES_EXECMY_SET_PAGES_EXEC(&paPages[iPage], 1);
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/alloc-r0drv-linux.c: In function ‘VBoxHost_RTMemContFree’:
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/the-linux-kernel.h:324:47: error: implicit declaration of function ‘set_pages_nx’ [-Werror=implicit-function-declaration]
 # define MY_SET_PAGES_NOEXEC(pPages, cPages)  set_pages_nx(pPages, cPages)
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/alloc-r0drv-linux.c:493:13: note: in expansion of macro ‘MY_SET_PAGES_NOEXECMY_SET_PAGES_NOEXEC(&paPages[iPage], 1);
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/memobj-r0drv-linux.c: In function ‘rtR0MemObjLinuxVirtToPage’:
/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:922:27: error: passing argument 1 of ‘pud_offset’ from incompatible pointer type [-Werror=incompatible-pointer-types]
     u.Upper = *pud_offset(&u.Global, ulAddr);
In file included from /mnt/gentoo/usr/src/linux/include/linux/mm.h:70:0,
                 from /var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/the-linux-kernel.h:98,
                 from /var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:31:
/mnt/gentoo/usr/src/linux/arch/x86/include/asm/pgtable.h:826:22: note: expected ‘p4d_t * {aka struct  *}’ but argument is of type ‘pgd_t * {aka struct  *}’
 static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
cc1: some warnings being treated as errors
make[4]: *** [/mnt/gentoo/usr/src/linux/scripts/ /var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/alloc-r0drv-linux.o] Error 1
make[4]: *** Waiting for unfinished jobs....
cc1: some warnings being treated as errors
make[4]: *** [/mnt/gentoo/usr/src/linux/scripts/ /var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/memobj-r0drv-linux.o] Error 1
make[3]: *** [/mnt/gentoo/usr/src/linux/Makefile:1512: _module_/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv] Error 2
make[3]: Leaving directory '/mnt/gentoo/usr/src/linux'
make[2]: *** [Makefile:152: sub-make] Error 2
make[2]: Leaving directory '/mnt/gentoo/usr/src/linux'
make[1]: *** [Makefile:304: vboxdrv] Error 2
make[1]: Leaving directory '/var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv'
make: *** [Makefile:41: all] Error 2

This is strictly an upstream bug. Do not report it to the Gentoo devs. It’s not a Gentoo problem.

Let’s get into what exactly happened here. The quick and dirty explanation is: Linux uses a page table to map virtual memory addresses to physical memory addresses in an efficient way. A quick explanation can be found here. For a little over a decade, this table has been four “levels” deep. Today, we’re beginning to hit the limit of how much memory this will allow us to address, so Intel has concocted a 5-Level paging spec. The catch is, the new level was added between two other levels, not just tacked onto the end, so adjusting dependent code isn’t as simple as just doing one extra fetch (from what I’ve seen at least).

The Oracle devs are aware and a bug exists, 16725. They’ve actually already fixed this in the development version of their source code. So you can work around this issue in the meantime via the following user patch:

8 responses on “Bleeding on the Edge: app-emulation/virtualbox-modules-5.1.22 vs. kernel 4.12.0-rc2

  1. I’ve added the patch from github to the virtualbox-modules-5.1.22.ebuild and tried emerging it (with kernel 4.12.0 final)… It still fails 🙁

    I’m pretty sure that the patch is applied correctly, because the files in /var/tmp/portage/app-emulation/virtualbox-modules-5.1.22/work/vboxdrv/r0drv/linux/ now contain the lines referring to kernel 4.12 from the patch.

    The relevant lines from the failing build processs seem to be:
    /usr/src/linux-4.12.0/arch/x86/include/asm/atomic.h: In function ‘atomic_try_cmpxchg’:
    /usr/src/linux-4.12.0/arch/x86/include/asm/atomic.h:192:2: error: undefined named operand ‘new’
    return try_cmpxchg(&v->counter, old, new);

    • That’s a new one! I don’t recognize this nor am I able to reproduce it even with the new challenge 4.12.0 final introduces. The fact that this is happening in the asm stuff makes me wonder if there might not be some incompatible CFLAGS. What is your processor and what are your CFLAGS and CXXFLAGS?

    • Indeed I am. Part of the motivation behind this post was that I had to get it working as it is a key part of my workflow for my job. But know that I’m only running RC7 at the moment. There seems to be some new issues with 4.12.0 final. Also be aware that I compile my own kernel. I’m not using sys-kernel/gentoo-sources.

  2. I’ve now gotten it to work by using the Virtualbox 5.1.23 testbuilds from

    To get them to work in Gentoo I copied the existing virtualbox-bin ebuild, named it virtualbox-bin.5.1.23.ebuild and replaced all the http download with links to the current testbuilds. I also had to copy and modify the virtualbox-modules ebuild and change the http download links to a self-created module file that I had created with the .sh file from here:
    (which is the normal download location for the virtualbox-modules script)

    Now the modules compile and my guest VM has started just fine.
    (I’m using vanilla-sources – 4.12.0)

    • Ah, that’s consistent with the comments by the Oracle devs in the bug linked in the post:

      Confirming that guest additions appear to build normally when using VBoxGuestAdditions_5.1.23-115855.iso.

      Glad you were able to meet with success!

Leave a Reply