Installing macOS 11 “Big Sur” Beta on Proxmox 6

Are you not a member of the Apple Developer Beta? If so, you can install a macOS Catalina VM instead!

This tutorial for installing macOS Big Sur Developer Beta using OpenCore has been adapted for Proxmox from Kholia’s OSX-KVM project and Leoyzen’s OpenCore configuration for KVM. You can get the full sourcecode on my GitHub here.

Requirements

I’ll assume you already have Proxmox 6.1 or 6.2 installed. Because Big Sur is currently in closed developer beta, you need to be part of Apple’s developer program, and so you will have a real Mac available to download Big Sur and fetch the OSK key.

Your Proxmox host computer’s CPU must support SSE 4.2, so for Intel your CPU must be at least as new as Nehalem, which was the first CPU generation to bear the “Core” i5/i7 branding. Older CPUs will cause the finder to repeatedly crash after installation completes (with an Illegal Instruction exception in the graphics code).

Modern AMD CPUs also support SSE 4.2 and will work with this guide.

First step: Create an installation ISO

Opt in to the Big Sur beta seed on Apple’s website, then you can use Software Update to download Big Sur:

https://developer.apple.com/macos/

Download this script, mark it as executable with “chmod u+x create_dmg_bigsur.sh” and execute it with “./create_dmg_bigsur.sh”:

https://raw.githubusercontent.com/thenickdude/OSX-KVM/master/OpenCore-BS/create_dmg_bigsur.sh

This will build a file called BigSurBeta.dmg on your desktop. Rename this file to “BigSurBeta.iso” so that we’ll be able to use Proxmox’s ISO picker to select it later, then upload this to your Proxmox’s ISO store directory (typically /var/lib/vz/template/iso).

Prepare an OpenCore image

Download the Big Sur-compatible OpenCore.iso.gz release from my repository, unpack it, and upload it to Proxmox’s ISO store at /var/lib/vz/template/iso. Although it has a .iso file extension, this is actually a hard disk image. This version of OpenCore has configuration changes that are required to boot the Big Sur installer successfully, you can’t use one of my older releases.

Fetch the OSK authentication key

macOS checks that it is running on real Mac hardware, and refuses to boot on third-party hardware. You can get around this by reading an authentication key out of your real Mac hardware (the OSK key). Save the first block of C code from this page as smc_read.c. In a command prompt, change into the same directory as that file and run:

xcode-select --install # If you don't already have gcc
gcc -o smc_read smc_read.c -framework IOKit
./smc_read

It’ll print out the 64 character OSK for you. Make a note of it.

Every Mac uses the same OSK, so don’t be surprised that it doesn’t look like a random string!

Create the VM

From the Proxmox web UI, create a new virtual machine as shown below.

In the Options page for the VM, ensure that “use tablet for pointer” is set to “Yes”.

In the Hardware page for the VM, add a second DVD drive at IDE0, set it to use your BigSurBeta.iso.

Don’t try to start the VM just yet. First, SSH into your Proxmox server so we can make some edits to the configuration files.

Edit /etc/pve/qemu-server/YOUR-VM-ID-HERE.conf (with nano or vim). Add this line, being sure to substitute the OSK you extracted earlier into the right place:

args: -device isa-applesmc,osk="THE-OSK-YOU-EXTRACTED-GOES-HERE" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check

A USB keyboard is added here because macOS doesn’t support QEMU’s default PS/2 keyboard. The CPU is set to pretend to be Penryn, because the macOS kernel patch that would support -cpu host has not yet been ported to Big Sur. Ensure the args are all on a single line!

Now find the lines that define the two “ISOs” (ide0 and ide2), and remove the “,media=cdrom” part from them. Add “,cache=unsafe” in its place. This will treat these as hard disks rather than DVD drives.

Save your changes, return to the Options tab, and change the boot order to put IDE2 (the OpenCore image) first. Your final VM configuration file should resemble this:

args: -device isa-applesmc,osk="..." -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check
balloon: 0
bios: ovmf
bootdisk: ide2
cores: 4
cpu: Penryn
efidisk0: vms:vm-100-disk-1,size=1M
ide0: isos:iso/BigSurBeta.iso,cache=unsafe,size=2094688K
ide2: isos:iso/OpenCore.iso,cache=unsafe,size=150M
machine: q35
memory: 4096
name: bigsur3
net0: vmxnet3=...,bridge=vmbr0,firewall=1
numa: 1
ostype: other
virtio0: vms:vm-100-disk-0,cache=unsafe,discard=on,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=...
sockets: 1
vga: vmware

Configure Proxmox

On Proxmox, run “echo 1 > /sys/module/kvm/parameters/ignore_msrs” to avoid a bootloop during macOS boot. To make this change persist across Proxmox reboots, run:

echo "options kvm ignore_msrs=Y" >> /etc/modprobe.d/kvm.conf && update-initramfs -k all -u

Install Big Sur

Now start up your VM, it should boot to the OpenCore boot picker:

OpenCore “OpenCanopy” boot menu

Press enter to boot the “Install macOS Beta” entry and the installer should appear.

Our virtual hard drive needs to be erased/formatted before we can install to it, so select the Disk Utility option. Follow the steps below to format the disk:

Now we’re ready to begin installation!

After the first stage of installation, the VM will reboot. Pick the “macOS Installer” entry (the second one here, with the hard disk icon) to continue installation:

The VM will reboot itself a couple more times, choose “macOS Installer” each time until it doesn’t reappear. Now the installation is complete, so choose the name of your main disk to boot (mine’s called Main).

Answer the initial install questions, and you’ll be logged on!

It works!

Make the OpenCore install permanent

We’re currently booting using OpenCore from the attached OpenCore ISO. Let’s install that to the hard drive instead. Pop open Terminal and run “diskutil list” to see what drives we have available.

Use “sudo dd if=<source> of=<dest>” to copy the “EFI” partition from the OpenCore CD and overwrite the EFI partition on the hard disk. The OpenCore CD is the small disk (~150MB) that only has an EFI partition on it, and the main hard disk is the one with the large (>30GB) Apple_APFS “Container” partition on it.

In my case these EFI partitions ended up being called disk1s1 and disk0s1 respectively, so I ran “sudo dd if=/dev/disk1s1 of=/dev/disk0s1” (note that if you get these names wrong, you will overwrite the wrong disk and you’ll have to start the installation over again!).

Now shut down the VM, and remove both the OpenCore and the Big Sur installer drives from the Hardware tab. On the Options tab, edit the boot order to place your virtio0 disk as the first disk. Boot up. If everything went well, you should see the OpenCore boot menu, and you can select your “Main” disk to boot Big Sur:

Sleep management

I found that I was unable to wake Big Sur from sleep using my mouse or keyboard. If you encounter the same problem, you can either disable system sleep in Big Sur’s Energy Saver settings to avoid the issue, or you can manually wake the VM up from sleep from Proxmox by running:

qm monitor YOUR-VM-ID-HERE 
system_wakeup
quit

Editing your OpenCore/EFI settings

The Configuration.pdf that explains the OpenCore config.plist file can be found along with the OpenCore release on my GitHub.

To mount your EFI partition in macOS so you can edit config.plist, first check the “identifier” of your EFI partition in the terminal:

~$ diskutil list
/dev/disk0 (external, physical):
   #:                   TYPE NAME              SIZE       IDENTIFIER
   0:  GUID_partition_scheme                  *512.1 GB   disk0
   1:                    EFI EFI               209.7 MB   disk0s1
   2:             Apple_APFS Container disk1   511.9 GB   disk0s2

Then you can mount it like so:

sudo mkdir /Volumes/EFI
sudo mount -t msdos /dev/disk0s1 /Volumes/EFI

Now you can edit /Volumes/EFI/OC/config.plist with your favourite text editor to make your changes. (TextEdit is not a great choice because it likes to insert curly quotes into the file and otherwise break things, there are some dedicated plist editors available such as XCode).

If you’re unable to boot macOS, you can edit the config.plist using the “UEFI Shell” option in the OpenCore boot menu instead.

Enter “FS0:” and press enter to open up the first available filesystem, then run “edit EFI\OC\config.plist” (if the file isn’t found, try switching to another filesystem like fs1:). When you’re done editing, press control+Q to exit, “Y” to save, then run “exit” to return to the OpenCore menu. You need to reboot for your changes to take effect.

Automatic boot

In config.plist, set Misc/Boot/Timeout to a non-zero value to allow the default boot option be chosen automatically after that delay in seconds. I’ve disabled this by default because it causes the installer ISO to re-enter its main menu instead of continuing the second stage of installation.

Verbose boot

To boot macOS in Verbose mode to diagnose boot problems, edit config.plist to change the “bootargs” in the NVRam section from “keepsyms=1” to “keepsyms=1 -v”.

Changing screen resolution

To change macOS’ screen resolution, you need to edit the UEFI/Output/Resolution setting in config.plist, the default is 1920×1080@32.

You should be able to change this to any of the modes that the system OVMF menu offers (hit F2 at the start of guest boot and choose “Device Manager/OVMF Platform Configuration” to see which resolutions are available).

Video performance

Because there is no guest video acceleration available for macOS, video performance is poor.

In Google Chrome in the guest you will need to toggle off the setting to “use hardware acceleration when available” to improve issues with elements not being drawn or flickering (especially video). Safari may be a better choice.

macOS’s built in “Screen Sharing” feature offers dramatically better framerates and latency than Proxmox’s browser-based VNC console, so if you have a real Mac to act as a viewing console, you can enable that in the VM’s “Sharing” settings and connect to the VM using the Screen Sharing app from your Mac instead:

Apparently Screen Sharing is also compatible with VNC clients like RealVNC, so you should be able to connect to it from Linux or Windows consoles using RealVNC.

The real magic bullet for video performance is to pass through a compatible video card using PCIe passthrough (though note that Big Sur, like Catalina, does not support most NVidia cards). This offers near-native performance. You can read more about how I’m using PCIe passthrough on my own installation here.

USB passthrough

Since I want to use this as my primary computer, I want to use a USB keyboard and mouse plugged directly into Proxmox, rather than sending my input through the web VNC console.

Proxmox has good documentation for USB passthrough. Basically, run “qm monitor YOUR-VM-ID-HERE”, then “info usbhost” to get a list of the USB devices connected to Proxmox:

qm> info usbhost
Bus 3, Addr 12, Port 6, Speed 480 Mb/s
Class 00: USB device 8564:1000, Mass Storage Device
Bus 3, Addr 11, Port 5.4, Speed 12 Mb/s
Class 00: USB device 04d9:0141, USB Keyboard
Bus 3, Addr 10, Port 5.1.2, Speed 12 Mb/s
Class 00: USB device 046d:c52b, USB Receiver
Bus 3, Addr 9, Port 14.4, Speed 12 Mb/s
Class 00: USB device 046d:c227, G15 GamePanel LCD
Bus 3, Addr 8, Port 14.1, Speed 1.5 Mb/s
Class 00: USB device 046d:c226, G15 Gaming Keyboard

In this case I can add my keyboard and mouse to USB passthrough by quitting qm, then running:

qm set YOUR-VM-ID-HERE -usb1 host=04d9:0141
qm set YOUR-VM-ID-HERE -usb2 host=046d:c52b

This saves the devices to the VM configuration for you. You need to reboot to have the new settings apply. Note that the emulated USB3 device doesn’t work with macOS, so don’t set “usb3=1”.

You can also pass through USB devices by passing through an entire USB controller using Proxmox’s PCIe passthrough feature, which gives much better compatibility.

FileVault

FileVault login prompt

FileVault now works with OpenCore, so you can encrypt your boot disk by using the option in macOS’ system security preferences. But be certain to keep a copy of your recovery key and keep your backups up to date.

Fixing “guest boots to UEFI shell”

If your guest ends up booting to the UEFI shell instead of showing the OpenCore boot menu, especially if you’ve just updated OpenCore to a new version, you’ll need to edit the guest’s UEFI boot entries to fix this.

At the very start of guest boot, hit F2 to enter guest UEFI settings.

First we’ll remove the old entries. Choose the Boot Maintenance option, then Boot Options -> Delete Boot Option. Use the spacebar to tick any old Clover or OpenCore entries (avoid ticking the EFI Internal Shell option, you want to keep that!). Select “Commit Changes and Exit”.

Now we’ll add the correct entry for OpenCore back in. Select Add Boot Option. Navigate through the device tree to EFI/BOOT/BOOTx64.efi and select it, name this new option “OpenCore” or similar. Again Commit Changes and Exit.

Go to the Change Boot Order and move OpenCore to the top. Commit Changes and Exit.

Now back out to the main menu and choose Reset, and you should successfully boot into OpenCore this time.

33 thoughts on “Installing macOS 11 “Big Sur” Beta on Proxmox 6”

  1. Great article. Thank you so much for posting this.
    I will try it today and I hope it works on my Proxmox as well.

    Greetings from Indonesia.

  2. Excellent article. Thanks. I cloned a Catalina VM and changed Clover to Opencore Boot, I used create_dmg_bigsur.sh and I have successful installed.

  3. Hi, Nick
    I am passing through my Radeon VII; when booting up, I can see the Apple logo and the progress bar for a while, then I get a little screen corruption and no signal afterwards. The system boots up fine, as I can remote into it through VNC, albeit the resolution is different than the regular VNC screen I get without the Radeon VII.

    This did not happen in previous Catalina installs.

    Should I try setting the resolution in OpenCore?

    Thanks!

    1. It might be outputting the image on a different port, try switching plugs on the back of the card. A visual glitch at this point in the boot is common, as the video card is initialised, but it shouldn’t turn off the screen afterwards…

      I’m assuming you’ve already set vga: none? The emulated video can mess things up if it is left enabled.

  4. I don’t see how you got Big Sur booting with the OpenCore config.plist you have provided. The require NVRAM settings are not in the config at all.

      1. The booter-fileset-kernel setting in NVRAM > Add section. This NVRAM setting is required in order boot Big Sur with prelinkedkernel support. Without this variable being set, Big Sur boots with KernelCollections.kc which is unable to be booted by OpenCore yet. I tried your Big Sur script and your OpenCore.iso with the exact same settings of your VM.conf settings and only got the same result of the SIGABRT loop with the installer.

        1. There must be something wrong with that info, because otherwise it could never have installed and been happily running for me (twice, as one VM I upgraded from Catalina and the other VM was a clean install). I’ve never had the setting you mention in any of my configs.

          I redownloaded my OpenCore image just now and used it to create a fresh new Big Sur VM, and it installed and ran fine. https://i.imgur.com/cJxROAr.jpg

          Post your VM config, and what’s your host CPU model?

          1. Host CPU is a AMD 3970x Threadripper, I have BigSur booting fine but it I had to use a real MacBook Pro with a NVMe to USB adapter and installed it to an external drive and then moved it to my Proxmox system. But I can boot the installer in any fashion.
            Here is my VM config
            “`
            args: -device isa-applesmc,osk=”…” -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt
            balloon: 0
            bios: ovmf
            bootdisk: ide0
            cores: 32
            cpu: Penryn
            efidisk0: local-lvm:vm-100-disk-1,size=4M
            hostpci0: 23:00,pcie=1,x-vga=1
            hostpci1: 01:00.0,pcie=1
            hostpci10: 48:00.3,pcie=1
            hostpci2: 02:00.0,pcie=1
            hostpci3: 43:00.0,pcie=1
            hostpci4: 44:00.0,pcie=1
            hostpci5: 46:00.0,pcie=1
            hostpci6: 47:00.0,pcie=1
            hostpci7: 48:00.1,pcie=1
            hostpci8: 4b:00.0,pcie=1
            hostpci9: 04:00.3,pcie=1
            localtime: 1
            machine: q35
            memory: 98304
            name: BigSur
            numa: 1
            ostype: other
            scsihw: virtio-scsi-pci
            smbios1: uuid=6623598a-7a98-4b99-8229-e44ed0d3568c
            sockets: 1
            tablet: 0
            vcpus: 32
            vga: none
            vmgenid: d3eb685a-544e-4936-b26a-89f3e0fec696
            “`

              1. There’s no point in trying to diagnose a VM with so many passthrough devices enabled because an incompatibility with any one of them could trigger a fault.

                Also I’d try reducing cores to 8 or 16 since it could be unhappy about the topology.

  5. Nevermind I just got the installer to boot, I recreated a new VM with the settings and it is working now, thanks for the settings.

  6. Hi Nick, I don’t have dev account, so I went for the BigSur ISO from the internet. But, I cannot see the BigSur logo/icon in the boot picker. I only see UEFI shell, Shutdown and Reset NVRAM.

    Do you think this is because of the ISO that I have? It is 10 GB size and it was used by someone to install it on VMware, and they said successful.

    Thank you again. I got my Catalina running. Now want to try this Big Sur

    1. OpenCore only supports disks formatted GUID/GPT, not MBR. There are a lot of images floating around that are formatted MBR and will be invisible in the OpenCore boot menu.

      If your image is a true ISO (rather than a harddisk image) then make sure you have retained media=cdrom in the config line for that drive. This isn’t going to be the case if the extension of your image is .dmg.

      From inside a Catalina VM you could run the Big Sur installer app from the image you have currently, which bypasses any boot problems since you never need to boot from the installer.

      1. Hi Nick, Finally I can get the BigSur from my Hackintosh. So I created the DMG and convert to ISO as your instruction.
        But upon boot, I still unable to see the BigSur in the boot picker.
        Which settings I should play with? I use your OpenCore for Big Sur.

        Thanks again

        1. Did you remove media=cdrom from the Big Sur ISO and add cache=unsafe in its place? It’s required.

          If you didn’t follow my instructions to create the DMG then it’s probably partitioned MBR instead of the GPT/GUID that OpenCore requires for the boot icon to become visible.

          1. Hi Hello,

            I got errors when building the dmg. I think this is why my ISO is not visible on picker.

            In the .sh command: I saw this line: in_path=/Applications/Install\ macOS\ Beta.app

            And my actual .app name is: Install macOS Big Sur Beta, not Install macOS Beta.

            Is this mismatch command that causes the error during dmg creation using the create_dmg_bigsur.sh?

            Thank you

  7. Sorry to bother again, just want to inform you about my error:

    sudo: /Applications/Install macOS Beta.app/Contents/Resources/createinstallmedia: command not found

    And then I try to manually execute that command with the correct path, I got this:

    dhani@iMac-Pro-2 Desktop % sudo /Applications/Install\ macOS\ Big\ Sur\ Beta.app/Contents/Resources/createinstallmedia –volume /Volumes/install_build –nointeraction
    Password:
    dyld: Library not loaded: @executable_path/../Frameworks/IAESD.framework/Versions/A/IAESD
    Referenced from: /Applications/Install macOS Big Sur Beta.app/Contents/Resources/createinstallmedia
    Reason: no suitable image found. Did find:
    /Applications/Install macOS Big Sur Beta.app/Contents/Resources/../Frameworks/IAESD.framework/Versions/A/IAESD: stat() failed with errno=20
    /Applications/Install macOS Big Sur Beta.app/Contents/Resources/../Frameworks/IAESD.framework/Versions/A/IAESD: stat() failed with errno=20
    zsh: abort sudo –volume /Volumes/install_build –nointeraction

    Any idea about this error?

    Thank you

    1. I would delete the app and redownload it.

      Which system are you running this on, a Catalina VM or like a really old Mac?

    2. It looks to be the spaces in the file name. I renamed the app to `Beta.app` and changed the in_path in the script accordingly and it works.

      1. I met another problem and the solution is to increase the disk size from 12g to 14g in the script. The new beta app must have increased in size, 13g might be ok but I just give it more than needed.

  8. Amazing! Spent a lot of time to make it work bare metal with Catalina+opencore with no success, but with your approach it worked flawlesly the first time! Thanks a lot!!!

  9. Thanks for great article clearly showing how would the new Big sur like. Always good to know before a light user step in.

    It’s a bit sad to know SSE2 is a must for Big sur, my macbook pro 2009 with core 2 due could not be working with Big Sur.

  10. hey nick, great work. i have a 3960x 24/48c i can get it to boot with 8/16 core how czn i change the topology to utilize all 24/48c ive been busting my head with this one if you can help out.

    thanks

  11. Hi Nick, finally I can install BigSur with your new script.
    I am using 2K monitor. Before, I use 1080 monitor and I can enable the mirroring display in macOS. I can use the 2K monitor as 2nd display but that is not convenient. The mirroring is my only preferred way I guess.
    But now since I have 2K monitor, and I enable the mirroring, I cannot get a good resolution for my Monitor. It uses the default Display max resolution (1080).
    Is there any way to fix this?

    1. I’m using an app called RDM to set custom monitor resolutions with my RX 580:

      https://github.com/avibrazil/RDM

      This allows you to pick HiDPI options even if macOS doesn’t offer them for some reason.

      I think if you mirror, then both monitors will have to be the same resolution though.

  12. Hi Nick, first thank you for all the work you do. It helped me get into Proxmox in a big way.
    I am now rocking 2 servers with several VMs each, Windows 10, 8, Linux and couple MacOS, HS, Catalina and Big Sur Beta 3.
    I wanted to let you know that now that Beta 4 is out I managed to do a clean install no problem following your guide after you made the update to the script.
    I though maybe it was good to have confirmation that the guide still works even after the new Beta.
    Thanks again.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.