ffmpegX and/or Toast 8 - Loss of video quality in converting to DVD

[bonespur]

Setup:
i run OS X 10.4.11 on an Intel iMac. I have Toast 8, ffmpegX, MkvMerge, QTSync and a few other video tools

Source Files
avi +idx +sub

Loss of video quality during conversions
I've used several different processes in attempts to convert the source files to VIDEO_TS such that they'll play (and show subtitles) in a DVD player that does not play Xvid/DivX.

Here are the several processes, with color/bold/underline text indicating video quality (black very good; blue poor; red very poor):
1. (avi +idx +sub) > mpg (with ffmpegX & mencoder) > VIDEO_TS (with Toast 8)
2. (avi +idx +sub) > mkv (with MkvMerge) > VIDEO_TS (with Toast 8)
3. (avi +idx +sub) > mkv (with MkvMerge) > XviD (with ffmpegX) > VIDEO_TS (with Toast 8)

In all case, the final video quality is extremely poor and unpresentable. The video quality of the final subtitles appears worse than the video quality of the rest of the final video.
In process 1, I selected all the best video quality options in ffmpegX.
In processes 1, 2 and 3, I selected the best video quality options in Toast 8.

Help, please.

[Case]

Please describe "poor" and "very poor"/"extremely poor". The way it looks may point to what went wrong and possibly how to solve it.

For better looking subtitles, you may want to convert (OCR) VobSub idx/sub to SubRip srt, using e.g. AviDemux, so the subs can be rendered anti-aliased with full color range (more than the 16 colors max (often less) of VobSub) and possibly set in a better looking font.
If you convert to .mpg with ffmpegX, you could make the output DVD compliant, so that Toast does not re-encode and will keep the current quality.
If you convert to 23.976 fps (NTSC FILM) framerate .mpg with ffmpegX, remember to "Set 3:2" pulldown, so that Toast doesn't re-encode to 29.97 fps.

Please be more precise about "all the best video quality options", as others (me included) are getting quite good results. Is Toast set to Automatic or Custom encoding? What was your video bitrate (in ffmpegX and Toast)? Did you select Half-PEL motion estimation in Toast? For mpg input in Toast, did you select 'Never reencode'? In ffmpegX, did you set Qmin=2?

Your process 3 should always look worse (in the details) than process 2, as there is a generation loss of one extra encode.

[bonespur]

Wow. Thanks for all the pointers.
Sorry I didn't note down all the settings I used in ffmegX.
Certainly I didn't do some/many of the things you've indicated.
i'll have to re-do, paying attention to the settings you've pointed to.
It'll take a while, but I'll add to the posting with results when I get them.
It's very good to hear that you and others do get good results.

Regarding "very poor"/"extremely poor", from subjective assessment I mean very noticeable loss of detail/crispness and I'd guess color quality too, plus the subtitles (which I burned into the video, and are no longer "soft") are very blocky and practically unreadable.

P.S.
While waiting for a response to the original posting, I found a quick alternative:

(avi +idx +sub) > mkv (with MkvMerge) > DVD NTSC, i.e. mpg (with Any Video Converter for Mac free version) > VIDEO_TS (with Toast 8 with DVD-Video/More/Encoding/Reencoding set to Never).

The video quality of the final video looks pretty good upon quick inspection in DVD Player. I suspect I did better for video quality this time because of producing the mpg files using Any Video Converter and also because of not reencoding in Toast 8. Alas, I still need to see whether I can re-do, in order to avoid/eliminate the black borders that appeared around the image in the videos outputted from Any Video Converter - currently I'm hoping I'll be able to do so by selecting "keep aspect".

Even so, I will still be re-doing the conversion using ffmpegX and Toast 8 in better ways as pointed out by Case above, as I wrote above that I will.
That should help me better understand ffmpegX and Toast 8 and improve my ability with them, including getting very good video quality when using those applications.