I would like to play 3D games in a virtual machine

Quick answer: don't.

VMware Server or ESX do not have 3D support. Not at all.

VMware Workstation 6.5, Player 2.5 and Fusion 2 have limited 3D support. They support 99% of DirectX 9c with shader model 2. Some old games may work well while many newer ones suffer fom bad performance.

There is a list of games and how they work under VMware virtualization here:

http://communities.vmware.com/docs/DOC-1287

Also see this Zamba's article on getting games running here:

http://www.scampers.org/steve/vmware/index.htm

If you are a die hard gamer, you might want to consider to dual boot, or if you are a Linux user, check out something like Cedega from http://www.transgaming.com/.