How to create an ESXi v4 Whitebox ISO with Windows Tools

Monday, July 27th, 2009 | Dennis D. Spreen | VMWare

This is a short tutorial on how to build a custom ESXi v4.0 Whitebox ISO with Windows Tools (and an Ubuntu-VMImage for VMWare Player) by integrating a new driver (RTL8169 network card) into the Community Driver Pack and integrating this pack into the ISO, too.

See “Create an ESXi v3.5 Whitebox ISO” for creating an ESXi v3.5 Whitebox.

1. Warning:
You need a 64bit CPU in order to run ESXi v4! Download Processor Check for 64-Bit Compatibility on the VMware site and determine whether your CPU is supported!
Btw: the Intel T2300 CPU is NOT a 64bit CPU…

2. Warning:
There are reports that the RTL driver may not work on all boards because of the new 64-bit portion of the ESXi driver model. Try ESXi 3.5 then. Or wait until a new full supported 64bit driver is made. Check vm-help forum regularly.


ESX Hardware Requirements
VMware ESX 4.0 will only install and run on servers with 64-bit x86 CPUs. Known 64-bit processors:

  • All AMD Opterons support 64 bit.
  • All Intel Xeon 3000/3200, 3100/3300, 5100/5300, 5200/5400, 7100/7300, and 7200/7400 support 64 bit.
  • All Intel Nehalem (no Xeon brand number assigned yet) support 64 bit.

Use this tools
UltraIso $29.95 USD / Free Trial
Ubuntu Server 8.10 for VMWare Player Free
Bitvise Tunnelier Free for individual use (or Putty, but Tunnelier is preferred)
WinImage US$ 30.00 / Free Trial

Let’s prepare by downloading the VMWare ISO, the Community Driver Pack and the new driver.

Download VMware ESXi 4 Installable
Download Community Unified oem.tgz v1.1.0
Download Realtek RLT8168 Driver oem.tgz package
Download Realtek RTL8169 Driver (if needed, try above first!)

1. Extract image.tgz from the VMware ISO with UltraIso

2. Use Tunnelier to log into Ubunto and create a new directory called “esxi” (either with SFTP or by command console with mkdir esxi)

3. Move image.tgz to this esxi directory (use Tunneliers SFTP)
Extract the contents by changing to this directory
cd esxi
tar zxvf image.tgz

and remove the tgz file
rm image.tgz

4. Decompress the VMWare diskimage
cd usr/lib/vmware/installer/
bzip2 -d VMware-VMvisor-big-171294-x86_64.dd.bz2

Skip step 5-12 if you’re using the Realtek RLT8168 Driver oem.tgz package

5. Make a new directory for the Community Driver Pack
cd
mkdir oem
cd oem

6. Upload the Driver Pack into this directory and rename it to oem.tgz then extract the contents and remove the file with
tar zxvf oem.tgz
rm oem.tgz

7. Prepare a new directory for the realtek driver
cd
mkdir rtl8169

8. Upload the mymods-0.1.tar into this directory and extract the content with
cd rtl8169
tar zxvf mymods.tgz

9. Then add this driver to the pack
cp mod/r8169.o ../oem/mod
cd ../oem

10. Add the PCIID to the simple.map by using the texteditor nano (no, we are not using the “thing called vi”. Thanks).
cd etc/vmware
nano simple.map

11. Insert the line 10ec:8169 0000:0000 network r8169 at the appropriate place:
...
10df:fd00 0000:0000 storage lpfc_740
10df:fe00 0000:0000 storage lpfc_740
10ec:8169 0000:0000 network r8169
1166:0211 0000:0000 storage ide
1166:0212 0000:0000 storage ide
...

12. Now create the updated oem.tgz file and download it to windows
cd
cd oem
tar cfvz oem.tgz *

13. Now download the usr/lib/vmware/installer/VMware-VMvisor-big-171294-x86_64.dd file to windows and open it with WinImage (open partition0).

14. Use WinImage to replace the oem.tgz on partition 0 and save.

15. Transfer the dd file to ubuntu /esxi/usr/lib/vmware/installer/ and compress it
cd
cd esxi
cd usr/lib/vmware/installer/
bzip2 VMware-VMvisor-
big-171294-x86_64.dd

16. Now recreate the image.tgz file
cd
cd esxi
tar cfvz image.tgz *

17. Download image.tgz to Windows

18. Use Ultraiso to replace image.tgz and add oem.tgz (yes, as of v4 the file isn’t here by default as it was with v3.5)

19. Burn ISO and start installing ESXi.

ESXi Tips and Tricks
http://www.vm-help.com/


Tags: , ,

65 Comments to How to create an ESXi v4 Whitebox ISO with Windows Tools

alex
August 4, 2009

Is the Realtek RLT8168 Driver oem.tgz package
compatible with esxi 4.0 64bits ???

Thanks

loris
August 4, 2009

I tryed add RLT8168 Driver using http://www.icjacobsen.com/dld/oem.tgz and following your procedure but installation from cd failed for D945GCLF2D / ATOM330

Dennis
August 5, 2009

See “2. Warning” on top. Some claim to have it working under ESXi 4, some report it does not work. Looks like we need an updated 64 bit driver now. Check the vm-help forums regularly!

Dennis
August 5, 2009

btw, I’m running the esxi 3.5 server with an Intel T2300 cpu, so no esxi4 for me (on the server with the rtl card), on the other esxi servers I’m using Intel network cards…

Linux_noob08
August 16, 2009

I’m a bit confused regarding step 5-12. The driver pack is a *.oem.tar flie. In linux I just rename things and it still works?

at 8) I assume mymods-0.1.tar is the driver file from Realtek(for example)? Tried it but can’t find any r8169.o or .o-file.

It would be nice if you could make 5-12 a bit more detailed.
/thanks

Dennis
August 17, 2009

@linux_noob08: no. a *.tar file is a compressed file, an *.o file is a DLL equivalent file, thus just renaming doesn’t work.
just type the commands written above in that order and it should work (well, we need a new 100% compatible rtl 64bit driver, the old one may fail).
If you have any problems just contact me via email and I’ll help you.

Linux_noob08
August 17, 2009

Normally there is a email address hidden somewhere, but sadly I can’t find yours.

Problem is simple:
-Driver package from Realtek is r8168-8.013.00
-The Linux computer I’m working on is not the comp I’ll use the Vmware esxi on. I’ll just create the CD and then boot the esxi installlation on the server, which uses a realtek 8168c/8111c NIC. Checked that twice.
-The Linux comp uses an Intel driver, I just downloaded the Realtekdriver in hope to find the r8168.o file as mentioned in step 9. Only a r8168.h file exist. That is where I’m stranded.

How do I create/make/download the .o version of the 8168-driver for step 9? I tried compiling according to the readme and downloaded missing Kbuild/gcc/make packages. Still no progress. I’m such a noob.

loris
August 20, 2009

I’ll try version i found here then i’ll report
http://www.vm-help.com/forum/viewtopic.php?f=8&t=112

loris
August 20, 2009

Sorry it was an old link.
This is the correct one
http://www.vm-help.com/forum/viewtopic.php?f=17&t=697

loris
August 22, 2009

Install failed on a Atom 330 – Intel D945GCLF2D with r8168

Dennis
August 23, 2009

@loris: thanks for your effort!

Sacx from RandomBugs
August 25, 2009

If you are using http://www.icjacobsen.com/dld/oem.tgz with an ESXi 4.0 the you should move the pci.ids from /usr/share/hwdata/ to /etc/vmware, if not it will fails with a nice PSOD (Pink Screen Of Death) 🙂

Regards

Mario Simoes
August 29, 2009

Hello, Denis,

Doesn’t the driver have the be signed for this to work?
Regards,Mario

loris
September 1, 2009

I tryed putting r8168.o file in mod/ directory and pci.ids, simple.map files in etc/vmware/ directory of oem.tgz
Boot failed. I found in log file: Vmkctl Warning Loading module r8168 Failed. “Invalid module name r8168”
What’s wrong?

Oswaldo
September 12, 2009

Hi there, thanks for this guide … one suggestion, you might want to use the windoze version of bzip2 (http://gnuwin32.sourceforge.net/packages/bzip2.htm) to reduce time transfering the files back and forth to linux.
regards

malle
September 14, 2009

thx olswalo. stfp is so slow, using ubuntu live cd to make changes.
suggestion too: winrar can unzip bz2.

im tryin to get esxi run on intel essential. i have a d-link dge-528t which is similar to rtl8169, will hope is works 😀 i git the d-link card run with x64 drivers from realtek webseite, but i had to change driver.inf for win server 2003 x64, d-link does not provide x64 server drivers, very bad !

malle
September 14, 2009

HOOOOOOOORAH it works with modified oem.tgz. thanks ! no lvmdriver load error any more.

loris
September 14, 2009

Could you help me getting it working too?

Malle
September 15, 2009

ha sorry git another module error, it seems that r8169 module does not sopprt my dlink vender id ( 1186:4300 ) i put it in simple.map and pci.ids, but it seems that it has to be written into the module too. in 2.6 kernel source it is, but not in the vmware esxi module it seems. so i will buy online a used intel card. sorry

Dennis
September 21, 2009

malle: thanks for your investigation though!

asoncodi
September 23, 2009

To save transfer times, the tar/compress steps can be performed under Cygwin. http://www.cygwin.com/

kazso
September 30, 2009

This will never work. Basically you just integrate 32-bit drivers (Community Unified oem.tgz, R8168 oem.tgz) into a 64-bit system (ESXi 4). I wonder if anyone tried this out before posting this guide here…

Dennis
September 30, 2009

You’re somewhat right – see my “2. Warning”. I’ve read that some used the 64 bit drivers and it worked (could’nt verify since I’m not using those cards on an esxi4). The procedure to include a driver is written in here – you only need the 64bit driver – see “Check vm-help forum regularly” above.

Dave McLeod
November 5, 2009

Hi
I tried the steps here and got a checksum error boot failed press any key to continue…

I have a Gigabyte board GA-MA78GM-US2H board with
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]

I installed VMware workstation on Centos 5, thats how I got the SATA controller using lspci.

I then copied the ahci.o file from the Centos 5 and added it according to the steps above.

the iso I created refuse to boot with the error above.

Please help!!

Thank you

joker
December 16, 2009

Hi guys, I’ve succeeded to boot ESXi 4.0 on Intel DG41RQ board (realtek 8111D onboard) using this: http://www.vm-help.com/forum/viewtopic.php?f=12&t=1272

Thanks for this guide!

Bane
December 17, 2009

Just wanted to mention that this whole process can be done on windows by using bzip for windows available at:

http://gnuwin32.sourceforge.net/packages/bzip2.htm

Nime
January 27, 2010

Can anyone make a patch for ESXi 4.0 Update 1 with R8169, there is a free patch maker for Windows at http://download.cnet.com/Patch-Maker/3000-2383_4-10038161.html?tag=mncol

It’s much easy than publishing a how I did article. Thanks.

Zinou
February 12, 2010

Hi all,
I”ve tried to install ESXi4 on Intel core 2 duo E8400 3ghz, 4G RAM, Realtek RTL 8139/810X but always I get error message :
“No compatible network adapter found please consult Hardware Compatibility Guide for supported adapters”

I’ve tried with “Realtek RLT8168 Driver oem.tgz package” given in this site but no result.

Can any one please give me a link where can I find the right driver for my network adapter?

KR,

Zinou.
;

proxyd
February 13, 2010

And I have a similar problem, I will be very grateful for the help.

satish
February 25, 2010

Anyone else seeing the lvmdriver issue? With the above realtek oem.tzg I no longer hit the “No compatible network adapter ..” but now I see

lvmdriver load failure

I’m running on an Asus intel i7 quadcore with

Disk: Hitachi HDT721010SLA360 ATA Device
NIC: Realtek RTL8168C(P)/8111C(P) Family PCI-E Gigabit Ethernet NIC (NDIS 6.0)

Some of the newsgroups say that it is still a nic issue. Any suggestions or leads would be appreciated. I’ve tried with esxi3.5 and lots of permutations and combination.

Please bare in mind I’m relatively new to vm world. Have only brought up one old dell in the past (with hardly any issues).

Thanks,
Satish

mahan
March 9, 2010

i do all the things here but still no compatible network adapter find
please help me
thanks alot

Rimidalv
March 26, 2010

I followed this process steb by step using same version. If I try to install ESX4i int VMware WKS to test it, I got “Failed to find boot partition”

Dstarr
March 26, 2010

Slightly confused by steps 13 & 14. Does one add the ome.tgz to the VMware-VMvisor-big-171294-x86_64.dd and then save the updated file

i.e:
\
ldlinux.sys
mboot.c32
oem.tgz
safeboot.c.32
syslinux.cfg

Dstarr
March 26, 2010

Please nuke comment above, I did not realise there were four partitions once VMware-VMvisor-big-171294-x86_64.dd has been opened in WinImage.

matt
April 13, 2010

checksum error.. you need a way to modify the isolinux checksum in order to do this..

if anyone is successful in building an ISO, please post a link or at the very least a .torrent file so we can all share it..

aniki
April 16, 2010

hello, anyone can share out this patched torrent, so we can help each other?

sleven
April 20, 2010

Could you put the iso file with realtek rtl-8168 ?

sleven
April 20, 2010

For download?

Simon
May 1, 2010

anyone ho has this a working iso for rtl-8168, i have tried this but i don’t work

thank you

Joseph
May 31, 2010

How are you able to connect to the Ubuntu server image you linked to with Tunnelier when it has no SSH server/daemon installed? What am I missing here?

Dan
June 3, 2010

Joseph, try typing sshd and see what the response is, normally it will give you a sudo apt-get etc…. line that tells you what to type to download and install the ssh daemon. Atleast it does on ubuntu 9.10. But before you can even get to that point you will need to set up your network card and edit the resolve.conf file for the dns. Once you can ping by hostname, then try to get your sshd install going. Upon finishing download, it should start right up and then you can use tunnelier to ssh (port 22) into it. You really dont have to go thru all that tho, you can just use the windows tools listed above in the comments.

Venkata
June 15, 2010

Has anybody succeeded in making the ESX4i work using a Realtek 8168 / 8169 / 81111 adapter.
I believe all of these are Giga bit adapters.

I found out that 7Zip can do all that we are trying to accomplish with Ubuntu.

All the other steps described can be followed.
7Zip does not know to store a tgz archive, the tar extension can possibly be replaced for a tgz, when creating the image.tgz file.

Anyways when I created it, I ended up with a PSOD.

The Community Unified oem.tgz is corrupt and wont open.. unless I am doing something wrong..

Just to confirm that it is just the Ethernet card that is stopping me from moving on, I borrowed a Broadcom PCIe card and the install progressed further, to the Installation stage.

Why would ESXi not have a driver for Realtek ?
(My hardware is Gigabyte EP43-UD3L with 4GB 800Mhz, 1TB SATA 3, DualCore E7500, Asus PLCIe Video, Built in Gig ethernet etc.,)

CK Lee
June 28, 2010

I finally got the DGE-528T to work with ESX3i u4. The card was using a Realtek 8169S but the oem.tgz driver does not have the PCI identifiers for it configured into its pci_tbl list. So I downloaded the open-vdriver project, followed the steps to install a Centos 3.9 environment to compile the r8169 source code and then change the r8169.c code to include the pci id: 1186:4300. Then I changed the oem.tgz file to include my new modified driver and also the simple.map file and now it detects the DGE-528T card.

Adrian
July 1, 2010

Hi CK Lee can you post the new oem.tgz that you created for DGE-528T, i have the same card and issues?

Thank you

Roger
August 4, 2010

Has anyone been successful in creating an iso file that will work with the Realtek NIC? I have tried the steps above, but have been unsuccessful. I guess I don’t know enough about the tools to complete this process. If anyone has an ISO file that I could burn to disc to get my machine working I would really appreciate it.

Aubrey Kloppers
August 18, 2010

Hi CK Lee and Adrian
I also have this card and can’t get it to work in 4.1 (I believe 4.0 uses the same driver structure)

Please could you make the driver available?

Cheers
Aubrey

Vicky
August 29, 2010

Hi Dennis the flashed USB drive with the dd worked fine and helped to boot the server detecting the Realtek 8168 however after creating as in your instruction it again fails to detect the Realtek 8168. Can you please explain part
“18. Use Ultraiso to replace image.tgz and add oem.tgz (yes, as of v4 the file isn’t here by default as it was with v3.5)”

Where do I add oem.tgz is it in root of the CD. Is it must as it doesn’t exists by default

Thanks in advance

Shawn
September 30, 2010

I’m trying to slipstream the ISO for ESXi 4.1 per these instructions. I don’t see an image.tgz within the ISO image. I see a cimstg.tgz and a imagedd.bz2. Has the structure of the Esxi 4.1 ISO changed drastically since 4.0? If so, what is the procedure for doing this for 4.1?

Thanks

Carlos
November 16, 2010

Shawn,

Installation image on ESXi 4.1.0 is imagedd.bz2, the imagedd.md5 ,needs to be updated.

Thel
January 5, 2011

Hi CK Lee can you post the new oem.tgz that you created for DGE-528T, i have the same card and issues?
Already try modify the pci-id and simple.map file,but does not work. Do you can help me ?

thank you very much

Cyril
January 31, 2011

Would it be nice to have the modified ISO posted somewhere? My understanding is that ESXi V4.1 is free anyway and a lot of people have a realtek network card so if the job is already done, and thank you very much for doing it, we might as well share it rather than having to redo the wheel again and again. Just a thought ;o)

Dennis
February 1, 2011

ESXi is “free” but copyright stays at VMware – thus you are not allowed to post the (modified) ISO somewhere else than on VMware.com. Sorry.

Cyril
February 1, 2011

I was affraid of that… any chance to have somebody uploading it to a private ftp site to workaround the posting issue? just asking ;o)

21st Birthday Ideas
March 4, 2011

Thanks for the info. Creating an ISO is such a helpful skill.

James

gnoma
May 23, 2011

I am trying to build esxi4.0 installation iso.
Tryed many oem.tgz files from internet, no success. Tryed to build my own as it says up, no success, because the driver pack from that link is damaged archive.

Then
I’ve did everything as it says with the ready oem.tgz (skipped step 5-12), And I got pink screen of death.
Then I did as Sacx from RandomBugs says about the moving of pci.ids file, installation went normal, but after reboot, still no interface loaded.

During installation, cat /var/log/messages | grep 8169 or grep 8168, and no output. Grep for vmnic1 (I have vmnic0 as intel onboard and it is running normal), still no results.
After installation, same search of /var/log/messages and still nothing. How do I find out why the kernel module doesn’t load?

Thank you!

gnoma
May 23, 2011

Sorry for the reply post, I forgot to write that my network card is with 8169 chip

Umarzuki
May 25, 2011

any update on this article for ESXi 4.1?

gnoma
May 25, 2011

Yes, and there comes another question…
Because ESXI 4.1 x64 has a problem with some intel motherboards and I can’t install it directly.
I can run it only by installind 4.0 and then update. If somehow I make that driver runs on 4.0, will it be affected when updateing to 4.1?
Thanks : )

Andy
May 27, 2011

Erm I am lost!
1. Extract image.tgz from the VMware ISO with UltraIso

I am using “VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso” and there is no file called “image.tgz”

Am I missing something here?

gnoma
May 28, 2011

Andy are you sure that this is the installable operating system, not just update from something to version 4.1?
if you doubt so, just put it in the CD and try to boot and install it : )

Gundam
August 3, 2011

Andy: Yes I’m having the same issue that you have.
Gnoma: Yes I was first suspect that as well. But later I tried to install this VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso on my VMWare and it did install properly. So this is a full version of ESXi 4.1 with update 1. And there is no more image.tgz! VMWare don’t allow me to down load ESXi 4.0, which I assume the image.tgz is still there?

Onyrmom
February 5, 2012

This is as good of a place to put my comments as any: Both ESX and ESXi really only need two files: simple.map and the .o file which is the actual driver. Simple.map is found in /etc/vmware, and the .o files are found in /usr/lib/vmware/vmkmod. At boot time, simple.map is read to match PCI vendor/device codes that are reported to the OS from the BIOS with drivers in /usr/lib/vmware/vmkmod. All other files are trivial, but there are a few things to note:

1: The /etc/vmware/pciid directory contains xml formatted files that contain information on the installed drivers. It’s a good idea to make a .xml file in here – just copy the format of a driver for a similar device (network to network, storage to storage, etc). Remember: the tag in this file refers to the driver name, not the driver filename. For example, the tg3 driver is called “tg3”, not “tg3.o”, even though the file is named “tg3.o”. This will cause the “invalid module name” error listed above. Simple.map, however should refer directly to the filename (i.e. r8168.o).
2: There is a pci.xml file in /etc/vmware that can be ignored. It’s used to simply provide human-readable descriptions of various PCI vendor/device codes.
3: There are other pci-related files in /etc/vmware, but they are autogenerated – you don’t really need to touch them. However, the pci.ids file does contain a list of PCI vendors and devices that could potentially be required by the OS, so you may add your device here if you wish. The format is pretty simple to follow.
4: After you add your driver and pci pointer, look in /var/log/boot-logs for helpful information. “sysboot-vmkernel-boot.log” contains information on devices being identified by the BIOS and referred to the OS. If you see your pci device code here, then you know it’s available to the OS. You can also use “lspci -vv” and “lspci -nn” to get this information. “sysboot.log” will tell you about the driver loading process itself. If the kernel has a problem loading your driver, you’ll see the error messages here.
5: Use “vmkload_mod ” to see if you can manually load your driver. If it works here, then it’s just typo/permission/path problems that are preventing your driver from being loaded. There are VMware article on manually configuring driver options:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1038247

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1017588

I use the “esxcfg-module” command to set my options. There is a driver option config stanza that can be created in /etc/vmware/esx.conf, occasionally bad option entries will cause your driver to fail to load. The sysboot log should tell you this.

6: Personally, I’m testing all of this with the Realtek 8168 gig driver. There are many versions of this driver out there – I’ve found five, all compiled differently. YMMV. However, a note for those who just want the NIC to work: many of the pre-made “oem.tgz” files you will find contain an AHCI driver as well. You likely do not want this if you’re making a whitebox- most current manufacturers are using the Intel IDE chipsets, not the Realtek one that requires this driver. Remove it if you don’t specifically need it. It shouldn’t interfere if the PCI ID for the Realtek IDE chip is not found, but it just makes things easier to debug.

7: The process above applies equally to ESX 4.0, 4.1 (all updates), and ESXi of these versions as well. The difference with ESXi, of course, is that you must prepackage your driver and simple.map file into an oem.tgz that you will copy onto your *already installed* ESXi box. Of course, ESXi won’t install without a NIC and Storage, but I’m sure you’ll figure a way around that 😀

For all you lazy kids who are looking for a premade install .ISO with the drivers installed….you’re wasting your time. Just make your own oem.tgz. It’s easy enough to figure out how oem.tgz is put together by just pulling it apart on your Windows box with 7-zip and Notepad++. Just find a pre-made oem.tgz for whatever and copy the format. You should find all the files I’ve discussed above in various subdirectories and archives.

Onyrmom
February 5, 2012

Oops: correction to above

simple.map does NOT refer directly to the filename. It *might* work if you do not have the driver XML file created, but assume it won’t.

Onyrmom
February 5, 2012

And one last one: don’t forget to make a manifest file in /etc/vmware/init/manifests. Again, just copy the format of the other files.

What is a manifest file?
http://blogs.vmware.com/vapp/2009/08/inside-the-ovf-package.html

But you do not need to go start generating SHA hashes quite yet. The drivers will load just fine, although you will get errors in your vmkernel logs about loading unsigned drivers.

In this situation, the manifest file simply makes the logical connection between the location of the driver.o file and the vmkernel driver loading process. Without it, the boot process has no idea where to go to get your “r8168” driver, for example.

Gabriel
August 29, 2012

Hello, what happens with ESXi 5.0
I need migrate to this version and I have a RTL8169 card that works fine in ESXi 3.5

Regards.
Gabriel.

About Dennis D. Spreen

I'm an avid programmer working on a variety of platforms in a variety of languages with a wide technical interest.

Search

QR Code

Categories