Installing VMware Server 2.0.2 with Linux Kernel 2.6.31-*-rt Ubuntu Studio 10.04

March 8th, 2011  |  Published in hacks  |  1 Comment

I’m working on a virtualized environment to run scalable instances of the Public Mapping Project app.

While the project offers an EC2 AMI, my boss wanted to run this on our own hardware, so we’re going to use VMware.  To develop the instance images, I wanted to install VMware Server 2.0.2-203138 on my notebook which is running Ubuntu Studio 10.04 with a 2.6.31-11-rt kernel.

The installer provided by VMware doesn’t work out of the box for Ubuntu Systems.  So, I followed the instructions in the Ubuntu Community VMware Server Documentation which instructs users to use a patching system developed by Radu Cotescu.  While this was easy to use and clearly documented, it didn’t work for me.  This is what happened:

ghing@geoffsnotebook:~/Downloads$ sudo ./raducotescu-vmware-server-linux-2.6.3x-kernel-71f8b66/vmware-server-2.0.x-kernel-2.6.3x-install.sh .You have VMware Server archive: 	VMware-server-2.0.2-203138.i386.tar.gzChecking for needed packages on UbuntuYou do have the linux-headers-2.6.31-11-rt package...You do have the build-essential package...You do have the patch package...Extracting the contents of VMware-server-2.0.2-203138.i386.tar.gzFound .tar file for vsock moduleFound .tar file for vmci moduleFound .tar file for vmmon moduleFound .tar file for vmnet moduleExtracting .tar files in order to apply the patch...Untarring ./vmware-server-distrib/lib/modules/source/vsock.tarUntarring ./vmware-server-distrib/lib/modules/source/vmci.tarUntarring ./vmware-server-distrib/lib/modules/source/vmmon.tarUntarring ./vmware-server-distrib/lib/modules/source/vmnet.tarTesting patch...Creating some simlinks for the newer kernels...ln: creating symbolic link `/usr/src/linux-headers-2.6.31-11-rt/include/linux/autoconf.h': File existsln: creating symbolic link `/usr/src/linux-headers-2.6.31-11-rt/include/linux/utsrelease.h': File existsApplying patch...Preparing new tar file for vsock modulePreparing new tar file for vmci modulePreparing new tar file for vmmon modulePreparing new tar file for vmnet moduleChecking that the compiling will succeed...Trying to compile vmci module to see if it worksPerforming make in ./vmware-server-distrib/lib/modules/source/vmci-onlyUsing 2.6.x kernel build system./home/ghing/Downloads/vmware-server-distrib/lib/modules/source/vmci-only/linux/driver.c: In function ‘LinuxDriver_Open’:/home/ghing/Downloads/vmware-server-distrib/lib/modules/source/vmci-only/linux/driver.c:363: error: implicit declaration of function ‘init_MUTEX’make[2]: *** [/home/ghing/Downloads/vmware-server-distrib/lib/modules/source/vmci-only/linux/driver.o] Error 1make[1]: *** [_module_/home/ghing/Downloads/vmware-server-distrib/lib/modules/source/vmci-only] Error 2make: *** [vmci.ko] Error 2There is a problem compiling the vmci module after it was patched. 🙁

I began to suspect that my problem could be related to the realtime kernel used by Ubuntu Studio. Googling, I found that other realtime kernel users were having problems installing VMware products.

This thread offers a description of the problem and a patch for another VMware project. Based on this I was able to create my own patch for the VMware server kernel module sources. I then modified Radu’s patch and was able to run his shell script to successfully install VMware server.

Relevant files:

To use, simply download my updated version of Radu’s patch and save it in the directory where you unarchived Radu’s installer scripts.