Create an ESXi Whitebox ISO with Windows Tools

Friday, March 13th, 2009 | Dennis D. Spreen | VMWare

This is a short tutorial on how to build a custom ESXi v3.5 Whitebox ISO with Windows Tools (and an Ubuntu-VMImage for VMWare Player) by integrating a new driver (RTL8169 network card) into the Community Driver Pack and integrating this pack into the ISO, too.

See “How to create an ESXi v4 Whitebox ISO with Windows Tools”  for creating an ESXi v4 Whitebox.

Use this tools
UltraIso $29.95 USD / Free Trial
Ubuntu Server 8.10 for VMWare Player Free
Bitvise Tunnelier Free for individual use (or Putty, but Tunnelier is preferred)
WinImage US$ 30.00 / Free Trial

Let’s prepare by downloading the VMWare ISO, the Community Driver Pack and the new driver.

Download VMware ESXi 3.5 Installable Update 3
Download Community Unified oem.tgz v1.1.0
Download Realtek RTL8169 Driver
Download Realtek RLT8168 Driver! New! oem.tgz package!

1. Extract install.tgz from the VMware ISO with UltraIso

2. Use Tunnelier to log into Ubunto and create a new directory called “esxi” (either with SFTP or by command console with mkdir esxi)

3. Move install.tgz to this esxi directory (use Tunneliers SFTP)
Extract the contents by changing to this directory
cd esxi
tar zxvf install.tgz

and remove the tgz file
rm install.tgz

4. Decompress the VMWare diskimage
cd usr/lib/vmware/installer/
bzip2 -d VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd.bz2

5. Make a new directory for the Community Driver Pack
cd
mkdir oem
cd oem

6. Upload the Driver Pack into this directory and rename it to oem.tgz then extract the contents and remove the file with
tar zxvf oem.tgz
rm oem.tgz

7. Prepare a new directory for the realtek driver
cd
mkdir rtl8169

8. Upload the mymods-0.1.tar into this directory and extract the content with
cd rtl8169
tar zxvf mymods.tgz

9. Then add this driver to the pack
cp mod/r8169.o ../oem/mod
cd ../oem

10. Add the PCIID to the simple.map by using the texteditor nano (no, we are not using the “thing called vi”. Thanks).
cd etc/vmware
nano simple.map

11. Insert the line 10ec:8169 0000:0000 network r8169 at the appropriate place:
...
10df:fd00 0000:0000 storage lpfc_740
10df:fe00 0000:0000 storage lpfc_740
10ec:8169 0000:0000 network r8169
1166:0211 0000:0000 storage ide
1166:0212 0000:0000 storage ide
...

12. Now create the updated oem.tgz file and download it to windows
cd
cd oem
tar cfvz oem.tgz *

13. Now download the cd usr/lib/vmware/installer/VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd file to windows and open it with WinImage (Partition0).

14. Use WinImage to replace the oem.tgz on partition 0 and save.

15. Transfer the dd file to ubuntu /esxi/usr/lib/vmware/installer/ and compress it
cd
cd esxi
cd usr/lib/vmware/installer/
bzip2 VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd

16. Now recreate the install.tgz file
cd
cd esxi
tar cfvz install.tgz *

17. Download install.tgz to Windows

18. Use Ultraiso to replace install.tgz and oem.tgz (yes, we need to replace this one here, too)

19. Burn ISO and start installing ESXi.

ESXi Tips and Tricks
http://www.vm-help.com/

What’s new
Realtek RTL8168 driver by Jens Jacobsen

Tags: , ,

50 Comments to Create an ESXi Whitebox ISO with Windows Tools

bejker
March 13, 2009

I folow this tutorial but my 2 realtek cards 8139 chip, still not recognized by Esxi:(

Dennis
March 13, 2009

have you tried pressing ALT-F1 (or F2) to gain access to the console? type “unsupported” (without quotes) and use “lspci -v” to take a look at the PCI ID of the realtek 8139.
The PCI ID is a pair of four digit numbers, the vendor ID and device. You can use ‘lspci -v’ or ‘cat /proc/pci’ to find this information.

It should be 0x8139, so replace step.11 line with
10ec:8139 0000:0000 network r8169

Maybe the same driver works for this model, too. Please report if it worked out or not. thanks!

bejker
March 14, 2009

I mistyped 8139, I meant 8169. Today I found one more Realtek NIC with chip 8168 and I’m going to try this procedure with that NIC. I’ll report the result. tnx.

Cosmix
March 14, 2009

Hello ! In step 3) perhaps the correct command is:
tar zxvf install.tgz
Otherwise you shoud insert another step :
gunzip install.tgz
tar xvf install.tar

Maybe in a future version more hardware will be supported by default !
Best regards !

Dennis
March 16, 2009

@Cosmix: You’re right. Should be tar zxvf, but works even without it as stated (tar recognizes the gz format). But for clarity I’ll change it, thanks!

DaveP
March 17, 2009

I tried to build R8168 driver – the older r1000 driver from their web site seems to be the closest match.
I removed the debug and devel cflags and managed to install and load the module, /var/config.log shows it loaded and named vmnic0.
But it doesnt show as a Management Network selection – anyone managed to get this driver to work? Its also 1MB which is strange.

Cosmix
March 18, 2009

Hello Dennis ! The new version is already there :
http://www.vm-help.com/esx/esx3i/Custom_oem.tgz/CommunityUnifiedDriverPack_v1.1.0_U3-123629.oem.tgz

Best regards!

Dennis
March 18, 2009

@Cosmix: Thanks, I’ve updated the tutorial community download.

Peter
March 26, 2009

Hi!
I tried to install an 8168 card but when I booting, the system couldn’t load the r8169 module. I got a ‘VmkCtl’ warning after that the system couldn’t load the vmfs3 module, lvmdriver module, and finally I got a “failed to find HD boot partition” message.
What’s wrong?

Jeremiah C.
March 29, 2009

Looks like the RTL8168 isn’t 100% compatible with the RTL8169, see the ubuntu doc.

Gabriel
April 7, 2009

I have an IBM x3440 ESXi U3 working with 1 NIC (on Board) and I want to add another 8169 chipset NIC.

I dont want install it again (many VMs are working)
Is it possible add the new card without installing ESXi?

Regards

Gabriel
April 8, 2009

Do you have the link of the page you say?

Dennis
April 8, 2009

run virtual infrastucture client, click on the left on your host (esx.local or something), use right menu “configuration”, Hardware: Network Adpaters. Does it show up? should be vmnic1 then

Dennis
April 8, 2009

@Gabriel: just plug it in. take a look at the configuration page – network adapters

Gabriel
April 8, 2009

Ah, ok
No the NIC is not showed, and it is not showed in esxcfg-nics -l command in ESXi console.

I see PCI in command lspci -v

Dennis
April 10, 2009

search /etc/vmware/esx.conf for any mac or vminc entries, remove some of them and restart (there is an article somewhere on the vmware forum, can’t remeber where but there is one).

[…] for the heads-up on the Planex NIC Create an ESXi Whitebox ISO with Windows Tools | The curse of Dennis D. Spreen KIV first cuz a bit lazy at the moment to do hacking. Intel […]

Dennis
April 10, 2009

Thanks to Jens Jacobsen there is a RLT8168/8111 driver – see download section on top! Works great.

steve friedland
May 29, 2009

I am not a Linux guru. I have a new laptop with an unsupported nic. What is the dummy version of adding the new nic to ESXI. The install goes great, just does not load nic. lspci – v sees the hardware. Please help, trying to avoid 100hrs of nightmare.

Dennis
May 29, 2009

steve: which nic do you have?

steve friedland
May 29, 2009

The card is a broadcom. I am reimaging the laptop with the Vista OS to confirm exact type. Dell site had too many options. lspci sees the hardware. I have tried vmkload_mod to try some of the defaults in the /mod folder but keeping getting an error. Is there a possibilty of being able to add the integrated nic? I will have the exact details shortly.

steve friedland
May 29, 2009

Dennis- the card actually shows up as a
“Marvell Yukon 88e8040 pci Ethernet fast controller”.

How would I create a esxi install for this card, or modify the install after?

I appreciate any help you have time for.

Dennis
May 29, 2009

steve: sorry, i have no informations about marvell drivers. if you are familiar with coding, you may adapt the existing linux driver… but i don’t know of any ready compiled marvell drivers. sorry.

rela
June 28, 2009

Dear Dennis,

your blog post is very nice. It’s the first one of all the posts from the VMware help forum and so on which is very detailed and good explained. I changed some things for my personal server but the solution by including the drivers for a clean installation is perfect.
I’m very pleased for your hard work by publishing this here.
It was the one and only post where it is described how to integrate the drivers completely into an iso without installing on a USB-Key. I think this is a bad solution and not for efficient work.

Thank you very much.

Best regards!

Dennis
June 29, 2009

thanks rela – and a lot prefer the pxe boot (which is far easier to handle if you need to test your own built system drivers) 😉

Javier
July 7, 2009

Hi! I have a RTL8139d chiset net card.
http://www.realtek.com.tw/downloads/downloadsView.aspx?Langid=1&PNid=6&PFid=6&Level=5&Conn=4&DownTypeID=3&GetDown=false#2

lspci -v show 10ec:8139.

How I can do a custom oem.tgz to use that card?

Thanks!!
Javier.

Dennis Spreen
July 16, 2009

They claim to have it working:
http://ultimatewhitebox.com/nic

Andrew Roe
July 25, 2009

Would you mind updating this procedure (the driver injection) for ESXi 4.0? I’m entry level linux guy at best and am having a hard time finding the correct locations within the new build.

John Wilson
July 25, 2009

I get the following from the oem.tgz from your link. What have I missed??

[root@dooku oem]# tar zxvf oem.tgz
etc/
etc/vmware/
etc/vmware/simple.map
etc/vmware/simple.map.bak
etc/vmware/support
mod/
mod/3w_9xxx.o
mod/ata_piixsc.o

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Dennis
July 27, 2009

@John: looks like the file is corrupt. try downloading again, if still not working contact me, I’ll send you the file via email.

Santo
August 4, 2009

Hi,

I have a Dell Vostro 410 with a Realtek RTL8168C(P)/8111C(P) NIC and am able to install ESXi 3.5 with this custom oem.tgz

But as soon as I install the vmware tools in my vm image (win2k3 enterprise), the network adapter is behaving flaky:
– IE6 gives “400 Bad Request” when trying to access a webpage (e.g. http://www.google.com)
– Browsing my local network with windows explorer through “My Network Places -> Entire Network -> …” does not work, but accessing a server directly through its ip address (e.g. \\192.168.1.200) does work from time to time, though it’s very very slow and often times out
– ping does work.
– nslookup does work
– tracert does work

Without the vmware tools installed (or by just uninstalling the vmxnet driver from the tools), the network works without any problems but unfortunately the speed is only 10Mbps without those tools

Anyone else experiencing this or any suggestions to solve the issue ?

Note: I prefer ESXi 4 above ESXi 3.5 but unfortunately there is no realtek driver available (yet) for v4 afaik.
Any ideas if/when extra drivers will become available ?

Bart
August 9, 2009

I am trying to follow the above instructions, but are stuck on step 2. Log in to Ubuntu with tunnelier. What IP address to use? I assume 127.0.0.1?
Userid / password user/user?
Whatever I try, I get “no connection could be made because the target machine actively refused it”.
Do I need to start a service in Ubuntu first??

Cory
August 10, 2009

Hello,
Trying to use the R8168 driver from oem.tgz by Jen and having the issues described with it. Are there any updated drivers that DON’T have this issue?
Thanks

Shadrach
December 29, 2009

i followed your steps but it is not working for me not only me most of the friends are facing some problem so could you please upload edited version iso file (realtek nic suppost iso) on rapidshare or megaupload site and give us the link

Luis Guerrero
January 21, 2010

Hi, same comment as Shadrach did.
Im newbie at Linux and at this job i cant spare 1 or 2 hours trying to set it up; jst cant!! there´s to much work to do.
It could save us a lot of time if somebody could help us uploading fixed ISO ESX installer for Realtek.

Thanks a lot for all the job, the step by step guide and efforts to share it with us. Maybe ill spend a couple of overtime hours this weekend.

Dennis
January 21, 2010

Due to copyright restrictions please do not post a “realtek esxi ISO” link here, thanks (and sorry).

Srinivas Kagitapu
February 7, 2010

Even after injecing the OEM.TZ for 8168, it was throwing the same LVM driver failed. After inspecting a little bit I found some of the lines were mis-aligned in the Simple.map and pci.ids files. The mis-alignement was found exactly where the 8168 reference was provided. When aligned the lines and used the iso again, it worked as it should be.

Thanks
Srinivas Kagitapu

Dennis
February 10, 2010

thanks for yor hint!

jon
March 19, 2010

if you dont have the time to wait for an intel pro 1000 card trendnet gig card pci TEG-PCITXR will work and its based on the 8169 realtek

Bernd
June 7, 2010

i’m experiencing the same trouble santos does. running esxi 3.5u5 on servere with realtek nic. everything works fine and linux distributions even perfectly. only on windowsVMs (win 7, xp) I cannot connect to the internet with any browser, although – ping, nlslookup and tracert all work fine. strange behaviour… error: http 400. if i remove vmware tools the error is gone but performance slows down. any clues?

Misc
June 9, 2010

@Brat
Hi, did you managed to get past step 2? I’m struggling with same problems.

Maybe someone else can shed some light on the subject and give us newbies 😉 more details for step 2? Or give us some educational links ;).
I tried searching for tutorials on Tunnelier but didn’t find any that would help me.

Any help would be greatly appreciated!

mehdi
September 4, 2010

hello anyone
what is mymods-0.1.tar?
where is it? i can’t find it
i have a vostro 1500 laptop and try to install esxi 3.5u5 on it. but the esxi can’t detect my NIC ‘Broadcom 440x with pci/id 14e4/170c’
can anyone help me?
i have linux and windows drivers of my nic!
thanks

Bobosh
September 10, 2010

This vSphere 4.1 setup is very limiting. I need to build a broadcom 440x support too. Too bad we can’t just get custom ISOs with our drivers built in, that would be too simple right?… Thanks Hyper-V.

Rodolfo
September 19, 2010

I tried to do with the installation esxi4 but a problem occurred at the beginning of the installation, the message appeared … “Image checksum error, sorry”

[…] “Create an ESXi v3.5 Whitebox ISO” for creating an ESXi v3.5 […]

Adrian
February 23, 2011

I try add RTL8139 in esxi3.5u5 but it still now work with esxi, I use the way what you said March 13, 2009, so please tell me if you have anyother way.Thank you~~~`

visualizacion creativa
October 16, 2011

Appreciating the hard work you put into your website and in depth information you provide. It’s nice to come across a blog every once in a while that isn’t the same old rehashed information. Fantastic read! I’ve saved your site and I’m adding your RSS feeds to my Google account.

visualizacion creativa
October 16, 2011

Hi! I’ve been reading your weblog for some time now and finally got the bravery to go ahead and give you a shout out from Atascocita Tx! Just wanted to mention keep up the great work!

About Dennis D. Spreen

I'm an avid programmer working on a variety of platforms in a variety of languages with a wide technical interest.

Search

QR Code

Categories