Running ESXi 4.1 on Old Hardware

I was having problems with my (ancient) storage server and thought to myself how convenient it would be if the hardware was running ESX and I could remotely view the console output.  So, I started down the path of seeing what was possible.

It was previously running a flavor of Ubuntu, so I SSH'd into the system and checked out what kind of processor I was running:

clutzer@storage:~$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 5
model name      : AMD Opteron(tm) Processor 250
stepping        : 10
microcode       : 0x3a
cpu MHz         : 2390.227
cache size      : 1024 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fdiv_bug        : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext 3dnowext 3dnow constant_tsc tsc_reliable nonstop_tsc hypervisor
bogomips        : 4780.45
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Whoa, AMD Opteron 250--cute little thing.  A quick Google search told me that people have had success running ESXi 4.1 with this chip, so off I went to download it.

The install of ESXi 4.1 was pretty smooth.  However, this CPU is so old that even though it's a 64 bit CPU, it's only able to emulate 32 bit guests.  Sigh, oh well.  I can live with that until it dies and I upgrade to a modern CPU.  After an install of ubuntu-12.04.5-server for i386, I came across my first stumbling block: how to give my VM direct access to physical drives that were previously used for my RAID5 array.

To make a long story short, it's doable, it's just not support intuitive.  To start out with, I gave the guest physical access to my old boot drive so that I could copy settings from my older 64 bit ubuntu server to my new 32 bit one.  First, SSH into the ESX host (you will have to enable SSH in order to do this.)  Then CD into the directory containing your guest VM:

~ # df -h
Filesystem                Size      Used Available Use% Mounted on
visorfs                   1.3G    259.7M   1023.1M  20% /
vfat                      4.0G      1.1M      4.0G   0% /vmfs/volumes/5436990b-f73b2f64-36c4-00e08140bfd8
vfat                    249.7M     88.5M    161.2M  35% /vmfs/volumes/c36ca674-6c8e9e4b-31cf-d40952c191b0
vfat                    249.7M      4.0k    249.7M   0% /vmfs/volumes/372be47d-3edc4952-48c6-d8f964e543d6
vfat                    285.9M    145.3M    140.6M  51% /vmfs/volumes/f57412e8-354f182d-35e2-275460791eb0
vmfs3                    51.0G     21.5G     29.5G  42% /vmfs/volumes/5436990c-a9311db3-960e-00e08140bfd8
~ # cd /vmfs/volumes/5436990c-a9311db3-960e-00e08140bfd8/
/vmfs/volumes/5436990c-a9311db3-960e-00e08140bfd8 # ls
Gateway Storage Server
/vmfs/volumes/5436990c-a9311db3-960e-00e08140bfd8 # cd Gateway\ Storage\ Server/

/vmfs/volumes/5436990c-a9311db3-960e-00e08140bfd8/Gateway Storage Server #

Then, the following magical command creates a passthrough VMDK that you can then add to your guest VM as if it were any other VM disk:

vmkfstools -r /vmfs/devices/disks/vml.010000000020202020202020202020202035514630435a4e56535433333030 host_sata_ST3300620A_5QF0CZNV.vmdk

/vmfs/volumes/5436990c-a9311db3-960e-00e08140bfd8/Gateway Storage Server # ls -l
-rw-------    1 root     root         1073741824 Oct  9 17:52 Gateway Storage Server-db289c96.vswp
-rw-------    1 root     root        21474836480 Oct  9 18:02 Gateway Storage Server-flat.vmdk
-rw-------    1 root     root               8684 Oct  9 17:34 Gateway Storage Server.nvram
-rw-------    1 root     root                483 Oct  9 15:18 Gateway Storage Server.vmdk
-rw-r--r--    1 root     root                  0 Oct  9 15:18 Gateway Storage Server.vmsd
-rwxr-xr-x    1 root     root               3013 Oct  9 17:51 Gateway Storage Server.vmx
-rw-r--r--    1 root     root                277 Oct  9 17:51 Gateway Storage Server.vmxf
-rw-------    1 root     root       300069052416 Oct  9 16:56 host_sata_ST3300620A_5QF0CZNV-rdm.vmdk
-rw-------    1 root     root                506 Oct  9 16:56 host_sata_ST3300620A_5QF0CZNV.vmdk
-rw-r--r--    1 root     root              35175 Oct  9 15:25 vmware-1.log
-rw-r--r--    1 root     root              27116 Oct  9 15:25 vmware-2.log
-rw-r--r--    1 root     root             101220 Oct  9 15:32 vmware-3.log
-rw-r--r--    1 root     root             133671 Oct  9 16:15 vmware-4.log
-rw-r--r--    1 root     root              78331 Oct  9 17:34 vmware-5.log
-rw-r--r--    1 root     root              74680 Oct  9 17:56 vmware.log

Next, using vSphere Client log into the ESX box, edit the guest VM settings and add an existing drive.  Then, from your guest VM, mount any partitions you are interested in and Voila, you have passed through a drive.

The following are some command lines I'm storing here for reference:

Old boot drive:
 vmkfstools -r /vmfs/devices/disks/vml.010000000020202020202020202020202035514630435a4e56535433333030 host_sata_ST3300620A_5QF0CZNV.vmdk

4 drive RAID-5 array:
vmkfstools -r /vmfs/devices/disks/vml.01000000002020202020202020202020203558573030583259535433313530 host_sata_ST31500541AS_5XW00X2Y.vmdk
vmkfstools -r /vmfs/devices/disks/vml.01000000002020202020202020202020203558573031334e30535433313530 host_sata_ST31500541AS_5XW013N0.vmdk
vmkfstools -r /vmfs/devices/disks/vml.01000000002020202020202020202020203558573031413841535433313530 host_sata_ST31500541AS_5XW01A8A.vmdk
vmkfstools -r /vmfs/devices/disks/vml.01000000002020202020202020202020203558573031444731535433313530 host_sata_ST31500541AS_5XW01DG1.vmdk

H.264 in the Home

Introduction

Well, I've spent a long time trying to figure out an effective way of archiving content. Here's the scenario: you've got a DVR, you've got Netflix, you've got home videos, how do you store it all? I've decided that an MPEG-4 is the way to go over Windows Media. The reason is simple, in the IPTV world, MPEG-4 and UDP streaming dominates. For the home I can argue it makes sense as well.

First, MPEG-4 compression is good. It'll eventually be as good if not better than WMV9 Advanced Profile and VC-1. Second, if you have any device that's not based on Micro$oft, most likely it will be able to play some kind of MPEG-4 variant. Examples: Quicktime and DivX are MPEG-4. Third, MPEG-4 encoders and players can be found for free (especially the open source ones).

Anyway, let me get right to it. I use H.264 for the following purposes:

• Backing up my DVDs
• Recording television shows

Let me explain how this is accomplished. But, first a note: this is a Linux solution. Why? Because my Media Center is MythTV (which is a Linux-based PVR). If you are using Windows, the procedure most likely is portable (for example, I know x264.exe exists, and I'm pretty sure I've seen ffmpeg.exe before). Now, onto the nitty gritty details.

Conventions

• # - means execute the command as the root user
• $ - means execute the command as a normal user
  

Backing Up DVDs

When I back up a DVD, I only store the main movie title. Remember DVDs can have many titles and chapters. I don't really want all the out-takes, deleted scenes, etc. If I do, then I buy the DVD ;-). So, step 1 is to "rip" the VOB file from the DVD. This means we take all of the chapters associated with the main movie title, decrypt them, and store them in a single VOB file on disk. There are several tools to do this (including one I wrote myself based on MythDVD). The most readily available one is vobcopy.

Prerequisites

The following tools are required:

# yum install vobcopy
# yum install transcode

Note: transcode brings ffmpeg with it, which is what we use for the transcoding. You may have to add freshrpms to your YUM repository configuration. Also potentially ATrpms. You should be able to Google and figure out how to add those. If not, email me and I'll post directions.

Make sure you have x264 installed:

# yum install x264

Big fat note: I had to recompile ffmpeg because the one that comes from freshrpms didn't have H.264 support built in.

Ripping & Transcoding

Now, using vobcopy, rip the main movie title. I am currently using Fedora Core 5 with the following vobcopy command:

$ vobcopy -l

That's an "ell" argument meaning large file support. This tells vobcopy to create a single, large output VOB file. You should be able to tell where the resultant VOB file gets created. So for the sake of an example, I'm working on transcoding "The Girl Next Door" while I write this, so I'll use it for illustrative purposes. Here is the resultant VOB file that got created:

$ ls -l GIRL*
-rw-r--r-- 1 mythtv mythtv 4.2G Apr 23 14:56 GIRL_NEXT_DOOR_UNRATED_1691.vob

Pretty big file eh? You can see how easy it is to chew up your entire hard disk storing DVDs in MPEG-2 Program Stream format (VOB).

Let's using a transcode tool and get familiar with what we are working with:

$ tcprobe -i GIRL_NEXT_DOOR_UNRATED_1691.vob
[tcprobe] MPEG program stream (PS)
[tcprobe] summary for GIRL_NEXT_DOOR_UNRATED_1691.vob, (*) = not default, 0 = not detected
import frame size: -g 720x480 [720x576] (*)
  aspect ratio: 16:9 (*)
    frame rate: -f 29.970 [25.000] frc=4 (*)
                PTS=0.1977, frame_time=33 ms, bitrate=9800 kbps
   audio track: -a 0 [0] -e 48000,16,2 [48000,16,2] -n 0x2000 [0x2000]
                PTS=0.1977, bitrate=448 kbps
                -D 0 --av_fine_ms 0 (frames & ms) [0] [0]
   audio track: -a 1 [0] -e 48000,16,2 [48000,16,2] -n 0x2000 [0x2000]
                PTS=0.1977, bitrate=192 kbps
                -D 0 --av_fine_ms 0 (frames & ms) [0] [0]
   audio track: -a 2 [0] -e 48000,16,2 [48000,16,2] -n 0x2000 [0x2000]
                PTS=0.1977, bitrate=192 kbps
                -D 0 --av_fine_ms 0 (frames & ms) [0] [0]
   audio track: -a 3 [0] -e 48000,16,2 [48000,16,2] -n 0x2000 [0x2000]
                PTS=0.1977, bitrate=192 kbps
                -D 0 --av_fine_ms 0 (frames & ms) [0] [0]

So, we've got one video track, and 4 audio tracks. That's how DVDs have the ability to play in English, French, Spanish, with Director's commentary, etc. We don't need all that; we only want the English track. Fortunately, the first audio track for DVDs in North America is the English one, so when we transcode, things tend to work out by default. But, if you want to save a different audio track instead of the first one, I recommend using tcdemux to remux the VOB file with only the specific audio track you need. I will do this for illustrative purposes by creating a new VOB file with only the English audio track.

$ tcdemux -i GIRL_NEXT_DOOR_UNRATED_1691.vob -A 0xe0,0x80 > GIRL_EN.vob
0=0xe0 1=0x80 2=0x0 3=0x0 4=0x0
$ ls -lh GIRL*.vob
-rw-rw-r-- 1 mythtv mythtv 3.7G Apr 23 15:19 GIRL_EN.vob
-rw-r--r-- 1 mythtv mythtv 4.2G Apr 23 14:56 GIRL_NEXT_DOOR_UNRATED_1691.vob

Notice how we saved about 500MB simply by stripping out the non-English audio tracks? Anyway, now if we tcprobe GIRL_EN.vob, you can see the simplified format:

$ tcprobe -i GIRL_EN.vob
[tcprobe] MPEG program stream (PS)
[tcprobe] summary for GIRL_EN.vob, (*) = not default, 0 = not detected
import frame size: -g 720x480 [720x576] (*)
  aspect ratio: 16:9 (*)
    frame rate: -f 29.970 [25.000] frc=4 (*)
                PTS=0.1977, frame_time=33 ms, bitrate=9800 kbps
   audio track: -a 0 [0] -e 48000,16,2 [48000,16,2] -n 0x2000 [0x2000]
                PTS=0.1977, bitrate=448 kbps
                -D 0 --av_fine_ms 0 (frames & ms) [0] [0]

In my mind anyway, this makes things a lot easier to work with. Oh, and I should mention, there is NO LOSS in running tcdemux to create GIRL_EN.vob. Remember a VOB file is a container format and we simply removed some of the contents, but we didn't actually do any video or audio conversion whatsoever.

Anyway, I think it's time to starting creating our H.264 content. With the toolset currently available, and the state of players on Linux and on Windows, it's my opinion that the AVI container format, H.264 video codec, and MP3 audio codec is the way to go. Now I should also mention that personally I'm trying to figure out a MPEG-2 Transport Stream format, H.264 video codec, and AC3 audio codec. The reason being that I want to be able to stream this content to an H.264 capable Set Top Box (STB) such as the Amino A124. So, I'll show how to create both formats here, but I have to point out that I can't figure out how to smoothly play the MPEG-2 TS format on XP or on Linux. Perhaps the TS output of ffmpeg isn't as good as it could be?

Format: AVI / Video Codec: H.264 / Audio Codec: MP3

Fortunately for us, ffmpeg rules, and this can all be done in one step. The only really important thing to decide on it the quality. I've come up with the following guidelines:

• Near-lossless: video bitrate: 5872kbps, audio bitrate: 128kbps (total 6Mbps)
• Excellent: video bitrate: 3872kbps, audio bitrate: 128kbps (total 4Mbps)
• Good: video bitrate: 2872kbps, audio bitrate: 128kbps (total 3Mbps)
• Fair: video bitrate: 1872kbps, audio bitrate: 128kbps (total 2Mbps)
• Poor: video bitrate: 1372kbps, audio bitrate: 128 kbps (total 1.5Mbps)
  

Use Near-lossless if you're an aficionado and have lots of hard disk space. Use Excellent if it's an action movie with lots of high-speed scenes, explosions, or just a lot of activity/movement. Use Good for girl movies, ha ha. Fair and Poor start to really show encoding artifacts. I would only use these if disk space is an issue, or you want something really small to watch on the plane, etc.

Anyway, let's make this happen. But not so fast, let's create a sample file that will encode pretty quick to give us a general idea of what the output will look like:

$ ffmpeg -i GIRL_EN.vob -f avi -b 2872 -vcodec h264 -g 300 -bf 2 -acodec mp3 -ab 128 -t 60 GIRL_EN-Good-sample.avi

This command can take a really long time. If you are transcoding at 2-4 frames per second, it won't finish for 8-12 times the length of the movie. Here's where CPU power really comes in handy. Let me explain a bit about what's going on in the above command:

• -i GIRL_EN.vob: should be obvious, specify the input file
• -f avi: force the output format to be AVI
• -b 2872: attempt to maintain a video bitrate of 2872 kbps
• -vcodec h264: use the H.264 video encoder (a la x264)
• -g 300: use a GOP size of 300
  
• -bf 2: use 2 b-frames (I have to understand this one more)
• -acodec mp3: use the MP3 audio encoder (a la lame)
• -ab 128: use an audio bitrate of 128 kbps
• -t 60: create a sample that's 60 seconds long
  
• GIRL_EN-Good-sample.avi: should be obvious, the output file

This encoding command is what's called as single-pass encoding. Most literature you'll read will tell you that doing multiple pass encoding produces better results. I agree, so here is what you really want to do (but it takes twice as long):

$ ffmpeg -i GIRL_EN.vob -f avi -b 2872 -vcodec h264 -g 300 -bf 2 -acodec mp3 -ab 128 -pass 1 /dev/null
$ ffmpeg -i GIRL_EN.vob -f avi -b 2872 -vcodec h264 -g 300 -bf 2 -acodec mp3 -ab 128 -pass 2 GIRL_EN-Good.avi

The differences between these commands and the previous one that generated the sample are:

• -t was removed because we wan't to transcode the entire movie now
• -pass 1 /dev/null: for pass 1 don't bother saving the output, write it to the null device
• -pass 2 GIRL_EN-Good.avi: for pass 2, save the output

Note: the x264 library writes out its own temporary log file that I could not override with the -passlogfile argument to ffmpeg. This sucks from a parallel transcoding point of view because this log file will potentially collide?

Anyway, once the above command completes, you should have an AVI file of your movie.

Format: MPEG-2 Transport Stream / Video Codec: H.264 / Audio Codec: AC3

We can use the same quality guidelines as defined in the previous section. Let's create the sample:

$ ffmpeg -i GIRL_EN.vob -f mpegts -b 2872 -vcodec h264 -g 300 -bf 2 -acodec copy -t 60 GIRL_EN-Good-sample.ts

The noteworthy differences here are:

• -f mpegts: create an output MPEG-2 Transport Stream
• -acodec copy: copy the source audio track as-is to the output file. For a DVD, this basically means we preserve the original Dolby 5.1, AC3 format
• .ts: use a .ts file extension denoting this file is a transport stream

I wouldn't try to play this file in Windows Media Player, I've never gotten a H.264 transport stream to properly play. VLC makes a better attempt, but doesn't play it smoothly. There is where my process breaks down, I don't really have a robust MPEG-2 TS solution yet. I'll update this when I do.

Conclusion

The future is definitely H.264. There is so much momentum behind it right now, I just don't see VC-1 catching up. Especially in Europe and South America where H.264 is spreading like wild fire.

I should have probably put this closer to the top, but I've create a Perl program to simply the transcoding step. Vob2h264.pl does the work of creating the sample, and doing the 2-pass encoding. The only thing you need to tell vob2h264.pl is the bitrate, and output format.

http://mandalore.dyndns.biz/svn/public/mythtools/helpers/vob2h264.pl

Please let me know if you have any questions, comments, or suggestions.

Christiaan's Climbing Trips

Christiaan's Climbing Trips