Create an ESXi Whitebox ISO with Windows Tools
This is a short tutorial on how to build a custom ESXi v3.5 Whitebox ISO with Windows Tools (and an Ubuntu-VMImage for VMWare Player) by integrating a new driver (RTL8169 network card) into the Community Driver Pack and integrating this pack into the ISO, too.
See “How to create an ESXi v4 Whitebox ISO with Windows Tools” for creating an ESXi v4 Whitebox.
Use this tools
– UltraIso $29.95 USD / Free Trial
– Ubuntu Server 8.10 for VMWare Player Free
– Bitvise Tunnelier Free for individual use (or Putty, but Tunnelier is preferred)
– WinImage US$ 30.00 / Free Trial
Let’s prepare by downloading the VMWare ISO, the Community Driver Pack and the new driver.
Download VMware ESXi 3.5 Installable Update 3
Download Community Unified oem.tgz v1.1.0
Download Realtek RTL8169 Driver
Download Realtek RLT8168 Driver! New! oem.tgz package!
1. Extract install.tgz
from the VMware ISO with UltraIso
2. Use Tunnelier to log into Ubunto and create a new directory called “esxi” (either with SFTP or by command console with mkdir esxi
)
3. Move install.tgz
to this esxi directory (use Tunneliers SFTP)
Extract the contents by changing to this directory
cd esxi
tar zxvf install.tgz
and remove the tgz file
rm install.tgz
4. Decompress the VMWare diskimage
cd usr/lib/vmware/installer/
bzip2 -d VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd.bz2
5. Make a new directory for the Community Driver Pack
cd
mkdir oem
cd oem
6. Upload the Driver Pack into this directory and rename it to oem.tgz
then extract the contents and remove the file with
tar zxvf oem.tgz
rm oem.tgz
7. Prepare a new directory for the realtek driver
cd
mkdir rtl8169
8. Upload the mymods-0.1.tar
into this directory and extract the content with
cd rtl8169
tar zxvf mymods.tgz
9. Then add this driver to the pack
cp mod/r8169.o ../oem/mod
cd ../oem
10. Add the PCIID to the simple.map
by using the texteditor nano
(no, we are not using the “thing called vi”. Thanks).
cd etc/vmware
nano simple.map
11. Insert the line 10ec:8169 0000:0000 network r8169
at the appropriate place:
...
10df:fd00 0000:0000 storage lpfc_740
10df:fe00 0000:0000 storage lpfc_740
10ec:8169 0000:0000 network r8169
1166:0211 0000:0000 storage ide
1166:0212 0000:0000 storage ide
...
12. Now create the updated oem.tgz file and download it to windows
cd
cd oem
tar cfvz oem.tgz *
13. Now download the cd usr/lib/vmware/installer/VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd
file to windows and open it with WinImage (Partition0).
14. Use WinImage to replace the oem.tgz
on partition 0 and save.
15. Transfer the dd file to ubuntu /esxi/usr/lib/vmware/installer/
and compress it
cd
cd esxi
cd usr/lib/vmware/installer/
bzip2 VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd
16. Now recreate the install.tgz file
cd
cd esxi
tar cfvz install.tgz *
17. Download install.tgz
to Windows
18. Use Ultraiso to replace install.tgz
and oem.tgz
(yes, we need to replace this one here, too)
19. Burn ISO and start installing ESXi.
ESXi Tips and Tricks
http://www.vm-help.com/
What’s new
Realtek RTL8168 driver by Jens Jacobsen
50 Comments to Create an ESXi Whitebox ISO with Windows Tools
I folow this tutorial but my 2 realtek cards 8139 chip, still not recognized by Esxi:(
March 13, 2009
have you tried pressing ALT-F1 (or F2) to gain access to the console? type “unsupported” (without quotes) and use “lspci -v” to take a look at the PCI ID of the realtek 8139.
The PCI ID is a pair of four digit numbers, the vendor ID and device. You can use ‘lspci -v’ or ‘cat /proc/pci’ to find this information.
It should be 0x8139, so replace step.11 line with
10ec:8139 0000:0000 network r8169
Maybe the same driver works for this model, too. Please report if it worked out or not. thanks!
March 14, 2009
I mistyped 8139, I meant 8169. Today I found one more Realtek NIC with chip 8168 and I’m going to try this procedure with that NIC. I’ll report the result. tnx.
March 14, 2009
Hello ! In step 3) perhaps the correct command is:
tar zxvf install.tgz
Otherwise you shoud insert another step :
gunzip install.tgz
tar xvf install.tar
Maybe in a future version more hardware will be supported by default !
Best regards !
March 16, 2009
@Cosmix: You’re right. Should be tar zxvf, but works even without it as stated (tar recognizes the gz format). But for clarity I’ll change it, thanks!
March 17, 2009
I tried to build R8168 driver – the older r1000 driver from their web site seems to be the closest match.
I removed the debug and devel cflags and managed to install and load the module, /var/config.log shows it loaded and named vmnic0.
But it doesnt show as a Management Network selection – anyone managed to get this driver to work? Its also 1MB which is strange.
March 18, 2009
Hello Dennis ! The new version is already there :
http://www.vm-help.com/esx/esx3i/Custom_oem.tgz/CommunityUnifiedDriverPack_v1.1.0_U3-123629.oem.tgz
Best regards!
March 18, 2009
@Cosmix: Thanks, I’ve updated the tutorial community download.
March 26, 2009
Hi!
I tried to install an 8168 card but when I booting, the system couldn’t load the r8169 module. I got a ‘VmkCtl’ warning after that the system couldn’t load the vmfs3 module, lvmdriver module, and finally I got a “failed to find HD boot partition” message.
What’s wrong?
March 29, 2009
Looks like the RTL8168 isn’t 100% compatible with the RTL8169, see the ubuntu doc.
April 7, 2009
I have an IBM x3440 ESXi U3 working with 1 NIC (on Board) and I want to add another 8169 chipset NIC.
I dont want install it again (many VMs are working)
Is it possible add the new card without installing ESXi?
Regards
April 8, 2009
Do you have the link of the page you say?
April 8, 2009
run virtual infrastucture client, click on the left on your host (esx.local or something), use right menu “configuration”, Hardware: Network Adpaters. Does it show up? should be vmnic1 then
April 8, 2009
@Gabriel: just plug it in. take a look at the configuration page – network adapters
April 8, 2009
Ah, ok
No the NIC is not showed, and it is not showed in esxcfg-nics -l command in ESXi console.
I see PCI in command lspci -v
April 10, 2009
search /etc/vmware/esx.conf for any mac or vminc entries, remove some of them and restart (there is an article somewhere on the vmware forum, can’t remeber where but there is one).
[…] for the heads-up on the Planex NIC Create an ESXi Whitebox ISO with Windows Tools | The curse of Dennis D. Spreen KIV first cuz a bit lazy at the moment to do hacking. Intel […]
April 10, 2009
Thanks to Jens Jacobsen there is a RLT8168/8111 driver – see download section on top! Works great.
May 29, 2009
I am not a Linux guru. I have a new laptop with an unsupported nic. What is the dummy version of adding the new nic to ESXI. The install goes great, just does not load nic. lspci – v sees the hardware. Please help, trying to avoid 100hrs of nightmare.
May 29, 2009
steve: which nic do you have?
May 29, 2009
The card is a broadcom. I am reimaging the laptop with the Vista OS to confirm exact type. Dell site had too many options. lspci sees the hardware. I have tried vmkload_mod to try some of the defaults in the /mod folder but keeping getting an error. Is there a possibilty of being able to add the integrated nic? I will have the exact details shortly.
May 29, 2009
Dennis- the card actually shows up as a
“Marvell Yukon 88e8040 pci Ethernet fast controller”.
How would I create a esxi install for this card, or modify the install after?
I appreciate any help you have time for.
May 29, 2009
steve: sorry, i have no informations about marvell drivers. if you are familiar with coding, you may adapt the existing linux driver… but i don’t know of any ready compiled marvell drivers. sorry.
June 28, 2009
Dear Dennis,
your blog post is very nice. It’s the first one of all the posts from the VMware help forum and so on which is very detailed and good explained. I changed some things for my personal server but the solution by including the drivers for a clean installation is perfect.
I’m very pleased for your hard work by publishing this here.
It was the one and only post where it is described how to integrate the drivers completely into an iso without installing on a USB-Key. I think this is a bad solution and not for efficient work.
Thank you very much.
Best regards!
June 29, 2009
thanks rela – and a lot prefer the pxe boot (which is far easier to handle if you need to test your own built system drivers) 😉
July 7, 2009
Hi! I have a RTL8139d chiset net card.
http://www.realtek.com.tw/downloads/downloadsView.aspx?Langid=1&PNid=6&PFid=6&Level=5&Conn=4&DownTypeID=3&GetDown=false#2
lspci -v show 10ec:8139.
How I can do a custom oem.tgz to use that card?
Thanks!!
Javier.
July 16, 2009
They claim to have it working:
http://ultimatewhitebox.com/nic
July 25, 2009
Would you mind updating this procedure (the driver injection) for ESXi 4.0? I’m entry level linux guy at best and am having a hard time finding the correct locations within the new build.
July 25, 2009
I get the following from the oem.tgz from your link. What have I missed??
[root@dooku oem]# tar zxvf oem.tgz
etc/
etc/vmware/
etc/vmware/simple.map
etc/vmware/simple.map.bak
etc/vmware/support
mod/
mod/3w_9xxx.o
mod/ata_piixsc.o
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
July 27, 2009
@John: looks like the file is corrupt. try downloading again, if still not working contact me, I’ll send you the file via email.
July 27, 2009
August 4, 2009
Hi,
I have a Dell Vostro 410 with a Realtek RTL8168C(P)/8111C(P) NIC and am able to install ESXi 3.5 with this custom oem.tgz
But as soon as I install the vmware tools in my vm image (win2k3 enterprise), the network adapter is behaving flaky:
– IE6 gives “400 Bad Request” when trying to access a webpage (e.g. http://www.google.com)
– Browsing my local network with windows explorer through “My Network Places -> Entire Network -> …” does not work, but accessing a server directly through its ip address (e.g. \\192.168.1.200) does work from time to time, though it’s very very slow and often times out
– ping does work.
– nslookup does work
– tracert does work
Without the vmware tools installed (or by just uninstalling the vmxnet driver from the tools), the network works without any problems but unfortunately the speed is only 10Mbps without those tools
Anyone else experiencing this or any suggestions to solve the issue ?
Note: I prefer ESXi 4 above ESXi 3.5 but unfortunately there is no realtek driver available (yet) for v4 afaik.
Any ideas if/when extra drivers will become available ?
August 9, 2009
I am trying to follow the above instructions, but are stuck on step 2. Log in to Ubuntu with tunnelier. What IP address to use? I assume 127.0.0.1?
Userid / password user/user?
Whatever I try, I get “no connection could be made because the target machine actively refused it”.
Do I need to start a service in Ubuntu first??
August 10, 2009
Hello,
Trying to use the R8168 driver from oem.tgz by Jen and having the issues described with it. Are there any updated drivers that DON’T have this issue?
Thanks
December 29, 2009
i followed your steps but it is not working for me not only me most of the friends are facing some problem so could you please upload edited version iso file (realtek nic suppost iso) on rapidshare or megaupload site and give us the link
January 21, 2010
Hi, same comment as Shadrach did.
Im newbie at Linux and at this job i cant spare 1 or 2 hours trying to set it up; jst cant!! there´s to much work to do.
It could save us a lot of time if somebody could help us uploading fixed ISO ESX installer for Realtek.
Thanks a lot for all the job, the step by step guide and efforts to share it with us. Maybe ill spend a couple of overtime hours this weekend.
January 21, 2010
Due to copyright restrictions please do not post a “realtek esxi ISO” link here, thanks (and sorry).
February 7, 2010
Even after injecing the OEM.TZ for 8168, it was throwing the same LVM driver failed. After inspecting a little bit I found some of the lines were mis-aligned in the Simple.map and pci.ids files. The mis-alignement was found exactly where the 8168 reference was provided. When aligned the lines and used the iso again, it worked as it should be.
Thanks
Srinivas Kagitapu
February 10, 2010
thanks for yor hint!
March 19, 2010
if you dont have the time to wait for an intel pro 1000 card trendnet gig card pci TEG-PCITXR will work and its based on the 8169 realtek
June 7, 2010
i’m experiencing the same trouble santos does. running esxi 3.5u5 on servere with realtek nic. everything works fine and linux distributions even perfectly. only on windowsVMs (win 7, xp) I cannot connect to the internet with any browser, although – ping, nlslookup and tracert all work fine. strange behaviour… error: http 400. if i remove vmware tools the error is gone but performance slows down. any clues?
June 9, 2010
@Brat
Hi, did you managed to get past step 2? I’m struggling with same problems.
Maybe someone else can shed some light on the subject and give us newbies 😉 more details for step 2? Or give us some educational links ;).
I tried searching for tutorials on Tunnelier but didn’t find any that would help me.
Any help would be greatly appreciated!
September 4, 2010
hello anyone
what is mymods-0.1.tar?
where is it? i can’t find it
i have a vostro 1500 laptop and try to install esxi 3.5u5 on it. but the esxi can’t detect my NIC ‘Broadcom 440x with pci/id 14e4/170c’
can anyone help me?
i have linux and windows drivers of my nic!
thanks
September 10, 2010
This vSphere 4.1 setup is very limiting. I need to build a broadcom 440x support too. Too bad we can’t just get custom ISOs with our drivers built in, that would be too simple right?… Thanks Hyper-V.
September 19, 2010
I tried to do with the installation esxi4 but a problem occurred at the beginning of the installation, the message appeared … “Image checksum error, sorry”
[…] “Create an ESXi v3.5 Whitebox ISO” for creating an ESXi v3.5 […]
February 23, 2011
I try add RTL8139 in esxi3.5u5 but it still now work with esxi, I use the way what you said March 13, 2009, so please tell me if you have anyother way.Thank you~~~`
October 16, 2011
Appreciating the hard work you put into your website and in depth information you provide. It’s nice to come across a blog every once in a while that isn’t the same old rehashed information. Fantastic read! I’ve saved your site and I’m adding your RSS feeds to my Google account.
October 16, 2011
Hi! I’ve been reading your weblog for some time now and finally got the bravery to go ahead and give you a shout out from Atascocita Tx! Just wanted to mention keep up the great work!
About Dennis D. Spreen
Search
Recent Posts
- How to compile Lua 5.4.0 for Android as a dynamic library using Android Studio 4
- Please make inline vars usable for production – fix RSP-28892
- How to compile Lua 5.4.0 as a Mac OS X dynamic library
- How to compile Lua 5.4.0 for Linux as a shared library
- How to compile Lua 5.4.0 for Windows
- Daily Wage – a Spigot/Bukkit plugin that pays out a daily wage
- How to compile Lua 5.3.5 for Windows
- Better Collada exporter for Blender with custom properties
- MOS6502-delphi – a MOS 6502 CPU emulator for Delphi
- Pass a multidimensional array as a parameter (with a hidden caveat)
Categories
Tags
Archives
- May 2020
- March 2020
- June 2019
- March 2017
- August 2016
- July 2016
- June 2016
- January 2016
- September 2015
- February 2015
- January 2015
- October 2014
- September 2014
- August 2014
- May 2014
- March 2014
- February 2014
- November 2011
- June 2011
- February 2011
- March 2010
- September 2009
- August 2009
- July 2009
- May 2009
- March 2009
- February 2009
- January 2009
- November 2008
- October 2008
- February 2008
- June 2007
Delphi Feeds
- How To Live Track The ISS International Space Station December 20, 2024
- SmartInspect 4 Beta 1 Released December 20, 2024
- Mastering FastReport in Delphi: Dynamic File Reporting and Visualization December 19, 2024
- This was 2024 overview December 19, 2024
- Mon bilan GitHub pour l'année 2024 December 19, 2024
- Re-Architecting Legacy Applications: Top Software Transformation December 18, 2024
- The Horror of finding the right database! Part 3 December 18, 2024
- Using TeeChart in Vector Magnetics’ RivCross Software for Precise Directional Drilling December 18, 2024
- Upgrading C++Builder 12.2: Tip #7, bcc64x tips! December 17, 2024
- Enhancing Delphi Code Safety: Best practices, tools, and context December 17, 2024
March 13, 2009