Both sections of this page are now out of date. If you want to move from VMware to VirtualBox, what you really, really want is VirtualBox 2.1 or later. As of that version, VirtualBox fully supports VMware's VMDK disk image format, so you can simply load your VMware disk image into VirtualBox, which makes the entire first half of this page obsolete. VirtualBox 2.0+ also features a much-improved networking system, which makes the second part of this page obsolete. So please consider this page as of historical interest only.
My tests with VirtualBox went well enough that I decided to move my webserver and mailserver VMs to it (from VMware Workstation 6 beta). VMware does the job great, but VBox is open source and a little faster. So the machine serving this page to you is now running in VirtualBox :)
The conversion process isn't exactly straightforward, though. Unfortunately there's currently no tools which can convert a VMware hard disk image into a VirtualBox one. So with help from the guys in #mdv-cooker IRC - especially Colin Guthrie, thanks Colin - I used the following method.
Basically we're going to keep the VMware machine running, create a new VirtualBox machine with an empty hard disk, boot it with a rescue CD, and rsync the entire system from the VMware machine to the VirtualBox one.
So, first of all, make the VMware machine as static as you can - stop all unnecessary services, etc. Just so no changes get lost in transition.
Now, run VirtualBox, create a new machine with whatever settings you like. Get the System Rescue CD image, and set that image as the CD drive for the VirtualBox machine. Boot it up.
Now set up the partitions on the hard disk - sysresccd has gparted, but I didn't use it, I just used fdisk, mkfs.ext3 and mkswap. It's obviously easiest if you're just going with a / and a swap partition, but you can adjust this for more complex cases. Once the partitions are done, mount them somewhere - I just used a single partition, and mounted it as /mnt/new .
You need to have rsync installed on both machines, and an ssh server running on the VMware machine and an ssh client on the VirtualBox machine. You also need to be able to transfer files as root from the VMware machine to the VirtualBox machine, so you must either set up keys appropriately or set the ssh server config to allow password-based login and allow direct root login. Obviously change these settings back to something more secure afterwards.
Once all that's set up, run this command:
rsync -avWHS -e ssh --exclude /proc --exclude /sys --exclude /dev --delete root@vmware:/ /mnt/new
where 'vmware' is the name or IP address of the VMware machine, and /mnt/new is wherever you mounted your new root partition on the VirtualBox machine. Then wait for a while. The entire machine will be rsynced across.
Now there's just some cleaning up to do. First, check /mnt/new/etc/fstab - I'd switched from hda1 as swap and hda5 as root to hda1 as root and hda5 as swap, so I had to adjust that. Also check /mnt/new/etc/lilo.conf for similar issues, and /mnt/new/etc/modprobe.conf (depending on what hardware you had set up in the VMware and what you have in the VirtualBox, you may have to change something here). Also, create the directories /mnt/new/proc and /mnt/new/sys (Mandriva at least will fail to boot if these don't exist).
Finally, you need to create the /dev tree and install the bootloader, and to do that, we have to chroot. Do
chroot /mnt/new. If it complains about /bin/zsh not being found, you don't have zsh installed on the VM - just copy the file from /bin/zsh to /mnt/new/bin/zsh and try again.
In the chroot, simply run
makedev /dev and then
lilo (assuming lilo is your bootloader, of course). Now everything is done. You won't be able to power down normally, so just shut off the machine via the VirtualBox menu. Unmount the CD image and boot up again, and the machine should boot just as normal.
The other thing I had trouble with was setting up host-based networking - bridged networking, effectively. As with VMware, you need this method of giving the VM a network connection if you want the machine to be accessible from the outside world (as I obviously do). The instructions in the VirtualBox manual don't work as printed, they seem to be incomplete. I'm very grateful to the author of this post on the Ubuntu forums, who figured out the whole process. You need the tunctl and brctl utilities. In Mandriva, they're in the tunctl and bridge-utils packages respectively. These are the commands you have to run, on the host machine, as root:
tunctl -t tap0 -u user (where user is the user who runs VirtualBox)
chmod 666 /dev/net/tun (or add user running VirtualBox to the 'tun' group, on MDV)
brctl addbr br0
ifconfig eth0 0.0.0.0 promisc (where eth0 is the interface to be bridged)
brctl addif br0 eth0
brctl addif br0 tap0
ifconfig tap0 192.168.1.50 up (where 192.168.1.50 is an arbitrary unused IP in the same subnet as your local network)
echo 1 > /proc/sys/net/ipv4/conf/tap0/proxy_arp
route add -host 192.168.1.5 dev tap0 (where 192.168.1.5 is the IP address of the host machine)
arp -Ds 192.168.1.5 eth0 pub (where 192.168.1.5 is the IP address of the host machine)
After that, I had to figure out how to create additional tap interfaces, as I want to run two VMs and they can't share. You don't need a second bridge, you can use the same one. These are the necessary commands:
tunctl -t tap1 -u user (where user is the user who runs VirtualBox)
brctl addif br0 tap1
ifconfig tap1 192.168.1.51 up (where 192.168.1.51 is an arbitrary unused IP in the same subnet as your local network, not the same as tap0)
echo 1 > /proc/sys/net/ipv4/conf/tap1/proxy_arp
route add -host 192.168.1.5 dev tap1 (where 192.168.1.5 is the IP address of the host machine)
Obviously you can adjust that set of commands to create a tap2, tap3 etc - as many as you need. Then you just assign one tap* interface per VirtualBox machine.
With these settings, if all goes well, the VirtualBox machine can get its own IP address on your local subnet, just as with VMware's bridged networking mode.
VirtualBox is currently only packaged for Cooker in Mandriva. As my VM host machine runs 2007.0, I backported it to 2007.0 for this setup. The packager doesn't want to put the current version into the official 2007.0 /backports repository as he doesn't consider it stable enough, but if anyone's interested in my backported packages (I also backported tunctl, which isn't in 2007 either), I can provide them on request.