I'm on a PSP - Firmware 5.00 (Released October 2008 - It's now April 2009) and trying to take advantage of the new features in the firmware:
(From the PSP Firmware history):
* The number of file types that can be played has been increased to include MPEG-4 AVC (H.264) Video Main Profile (AVC CABAC) in 640 x 480 pixels. Some video files of this type may not be playable.

I'd like, ideally to be creating video files that conform to this, so I can playback the video on more forgiving hardware (Nokia 5800 or on the mac) without having to have 2 seperate files.

I've tried following the settings for 720x480 VIDEO from a different post (https://forum.videohelp.com/topic326029.html#1961779),
"Under the VIDEO tab, choose:
Video codec - H264 [.MP4] (x264)
Video bitrate - 1213 kbit/s (I'm sure that this can be modified to a higher or lower amount, but I like the higher quality stuff)
Video size - 720x480
Autosize - unconstrained
Framerate - NTSC (29.97)
Under the AUDIO tab, choose:
Audio codec - AAC (MOV/MP4/3GP)
Audio bitrate - 128 kbit/s
Sampling - 48000 Hz
Under the OPTIONS tab, choose:
Use CABAC - ON
Constant bitrate - ON
Trellis (requires CABAC) - ON
Encoding profile - PSP
AVC level - 3.0
LEAVE ALL OTHERS AT THEIR DEFAULTS "

But, like many others, no joy, the file it produces being seen as "Unsupported data". I've tried moving it from the video file folder, giving it a special folder (which seems to have been a firmware 2 thing), renaming it with the M4V convention..no joy.

I can create files using much lower quality specs on FFMEPGX:
320x240, 512kbps, aac, 24000, 96kbps
but that's not really much cop on anything but the psp and looks awful watching something fast moving (presumably because the bitrate us 512). The sounds also pretty borderline.

Now, oddly, using PSP Video Convert 9 on Windows, I get a high quality working file with quite similar settings to the ones mentioned above;
Here's a working file's settings (as taken from Mediainfo Mac when I analysed the file):

General / Container Stream # 1
Total Video Streams for this File: 1
Total Audio Streams for this File: 1
Video Codecs Used: AVC
Audio Codecs Used: AAC LC
File Format: MPEG-4
Play Time: 21mn 13s
Total File Size: 175 MiB
Total Stream BitRate: 1154 Kbps
Title (Tag): testfile
Encoded with: Lavf52.31.0
Video Stream # 1
Codec (Human Name): AVC
Codec (FourCC): avc1
Codec Profile: Main@L3.0
Frame Width: 720 pixels
Frame Height: 480 pixels
Frame Rate: 23.974 fps
Total Frames: 30522
Display Aspect Ratio: 4/3
Video Standard: NTSC
Scan Type: Progressive
Colorimetry: 4:2:0
Codec Settings (Summary): CABAC / 3 Ref Frames
QF (like Gordian Knot): 0.123
Codec Settings (CABAC): Yes
Video Stream Length: 21mn 13s 105ms
Video Stream BitRate: 1 022 Kbps
Video Stream BitRate Mode: VBR
Bit Depth: 24 bits
Video Stream Size: 155 MiB (89%)
Date of Original Encoding: UTC 2009-04-27 11:07:34
Audio Stream # 1
Codec: AAC
Codec (FourCC): 40
Audio Stream Length: 21mn 13s 2ms
Audio Stream BitRate: 128 Kbps
Audio Stream BitRate Mode: VBR
Number of Audio Channels: 2
Audio Channel's Positions: L R
Sampling Rate: 48.0 KHz
Bit Depth: 16 bits
Audio Stream Size: 19.4 MiB (11%)
Date of Original Encoding: UTC 2009-04-27 11:07:34

And that looks beautiful on the PSP screen, and on the mac. Sounds good too - but then it would, the res is larger and the bitrate three times higher. I don't want to encode on the pc, of course, I'm a mac user! So I want to get FFMPEGX to produce this sort of quality, as the PSP can read it, we just need to get the settings right.

Maybe the comparison between the two will give someone here the information needed to know what to change in those original 3.30 firmwares posts settings to enable us to convert video in FFMPEGX (oh and I'm using version 0.0.9y, incidentally) in a higher quality and have it work.

I'm going to perform a load of trial runs with different settings (quicktime encoding on or off, resolution 640x480, NTSC movie, NTSC etc) and see if I can find a high res that works, but in the meantime, has anyone else had joy with getting higher res settings working than the 480x272 version from the other post? (Which wouldn't be much cop on a mac screen).