PGCEdit question: setting menu button highlight positions for letterbox

[Alex_ander]

I have a custom DVD authored by another person with menu buttons problem. The menu is 16:9 and the problem is with highlight area positions in letterboxed mode (they mismatch subpicture). When explored in DVDRemake, the buttons show highlight positions adjusted for pan/scan mode instead of letterbox (no vertical number correction). I can't even switch to letterbox button edit tab to do corrections manually. It is only possible to change 'letterbox' menu attribute to 'pan/scan' for matching menu subs but I don't want it (some details would move to overscan area).
My question is: is there an option in PGCEdit to (better automatically) set button highlight positions for letterbox mode, basing on numbers used in WS mode? Also, is there any software player to preview menu in letterboxed mode (most show WS where it looks OK, or stretch to 4:3 fullscreen)? Thanks.

[manono]

Hi Alex_ander,

I'm not entirely sure I understand you correctly, but, yes, you can adjust the button sizes and positions so they line up better with the underlying subpics. I'm assuming it was really set for Wide and Letterbox.

Open the DVD in PGCEdit and highlight one of the menus. Right-click it and hit "Menu Buttons". Near the bottom of the new screen you'll see Group 1 (Wide) ticked. If you only want to adjust the Letterbox buttons, change it to Group 2 and you'll see the picture change. You can do the same in the Menu Editor. Hit the "Edit" button at the bottom to get to the Menu Editor. You'll see Group 1 and 2 again near the top. Move the slider near the top to switch among the different buttons, and move the X position/Width/Y position/Height sliders to adjust the size and position of the buttons. It's actually easier to understand once you get in there and start playing around. When all done, "OK" and "Close", and back in the main screen. Then do the same for any other menus you have to change. After changing them all, File->Save DVD, and test it out. I've done this a bunch of times after converting PAL menus to NTSC menus and the process is pretty intuitive I think.

If you get lost or screw something up you can't fix, then go back to where you started with File->Backups->Restore Backups. Good luck.

[tinker]

Alex_ander
If you are talking about highlight box which show up in DVDRemake around the button, than you can adjust it to larger size as long as it covers your highlight in other ratio.
PgcEdit will do the same thing.
Your DVD player does not care how large or small it is as long as buttons get covered.
It does not matter if it will be in overscan area as it does not show on screen anyway.
If, however, highlites will be mismatched, than you will have to redo menu.
I do not know any software which will show letterbox properly.

[Alex_ander]

Thanks for your answers, manono and tinker. I'll try to do it with PGCEdit in the evening.

@ tinker Unfortunately, in my case it is not possible to set larger highlights for all modes: the menu uses a column of 15 narrow buttons (text and highlighted square pointers on the left), placed close to each other. Any vertical enlargement would cause highlighting of adjacent pointers. In DVDRemake (with this particular menu only) I can't get access to 'letterbox' button edit tab (only 'WS', 'pan/scan' and 'all modes' are shown while menu 'convert to 4:3' attribute is 'letterbox'). And even if I could get to that tab, I'd have to somehow calculate and type 2x15 pixel numbers for letterboxed highlights without opportunity to test the result visually. So I'll try to do it with PGCEdit where letterbox is previewable. If I understand it right, neither program can just take pixel numbers from WS highlights and calculate/set the corresponding numbers for letterbox mode.

[tinker]

I know you are familiar with DVDRemake, I just was not sure what the situation really is. It was not too clear from your first post.
Hope you will have a success with PgcEdit.

[Alex_ander]

In PGCEdit I have the same problem as in DVDRemake: group 2 of buttons is for pan/scan, group 3 disabled (grey). Type of conversion to 4:3 for VTS menu domain is editable (automatical letterbox or pan/scan or both), but group 2 is still pan/scan and group 3 unavailable. Could this be limited by subpicture properties? I was sure subpicture is resized (one of the 2 ways) together with background, but can there be separate streams for different modes or smth.? If nothing helps I'll simply set the whole menu to pan/scan. The preview shows correct highlight positions in that mode and nothing is overscanned. The menu editor is done very well in PGCEdit.

[tinker]

Alex_under
I really do not get what you want to do.
You have column of 15 buttons with highlights on left. To cover highlights you have to expand horizontally if any.
Letterbox will show same picture ratio on 4:3 TV as wide screen TV, only narrower to fit width of 4:3 screen, that is why black bars on top and bottom.
So really buttons should be in the same position as viewed on wide screen, if the menu was correctly created for wide screen.

[manono]

In PGCEdit I have the same problem as in DVDRemake: group 2 of buttons is for pan/scan, group 3 disabled (grey).

Then I'd say that's how the guy made the menus, as Wide/PS rather than Wide/Letterbox. To confirm, right-click the menu, hit "Domain Stream Attributes", and check what it says.

If you want to change them to Wide/Letterbox, I'm not quite sure how you'd do that, short of demuxing, remuxing (setting it for Wide/Letterbox), replacing using VobBlanker, and then adjusting the button positions and sizes. I fooled around the other day by just changing in PGCEdit from PS to Letterbox and it didn't seem to do anything. Maybe you can figure it out.

Group 3 isn't usually used. It's for when you have all 3, Wide/PS/Letterbox all at the same time. Or, in one case I saw, when Group 1 wasn't used at all, and Wide/Letterbox were Groups 2 and 3.

[Alex_ander]

Thanks for your support, manono and tinker.
Yes, that menu was originally authored for pan/scan conversion, but for some reason menu domain setting was letterbox (it is possible to change it to pan/scan to make highlights work). I also wanted to turn buttons to 'letterbox' for matching in style with a working menu in another VTS.
My main problem was that letterbox mode couldn't be used for button editing. Even without enlarging buttons I had partial highlighting of 2 pointers on 4:3 display.
After some reverse engineering I found a workaround in DVDRemake and seem to understand how all this actually works.
1.Conversion to other display modes data is stored independently for buttons and for menu domain (probably even separately: in ifo for menu domain and in menu VOB for buttons, excluding CLUT stored in ifo).
2.There can be 8 group id's for different display modes:

0 normal
1 wide
2 letterbox
3 wide, letterbox
4 pan/scan
5 wide, panscan
6 letterbox, pan/scan
7 wide, letterbox, pan/scan

Found this by exporting menu buttons to file (.mbl in DVDRemake), editing group id's and importing back to page instead of deleted buttons. Here's how a button is described in .mbl file:

{
groupId=1
colorScheme=1
autoAction=0
command=610400080088001F
links={1 1 0 2}
rect={93 232 251 269}
}

So if I leave 'letterbox' for menu domain and replace all 'groupId=4' with 'groupId=2' I get the desired 'letterbox' button edit tab on importing buttons back. Then I can either edit numbers or (better) export modified menu/ifo files and open the whole DVD in PGCEdit, where it is easier to edit highlights.
In future it would be nice to have display type settings for menu domain synchronized (in either software) with button display settings.

[tinker]

Thanks for shearing.
Sorry if I offended you with "letterbox" explanation, it was just thinking aloud.
This happen when you communicate via email.

[spyhawk]

Yes, that menu was originally authored for pan/scan conversion, but for some reason menu domain setting was letterbox (it is possible to change it to pan/scan to make highlights work).

Setting the video display attribute to letterbox was probably an authoring mistake, and it is easy to change to pan/scan display as you said it was originally authored this way. Besides, since there are only 2 button groups: wide and pan/scan, there is no letterbox button group for you to edit. As you found out, exporting the button info and modify the groupId allows you to edit the button rectangles/markings/hotspots in the Letterbox tab afterwards. The subpic highlight may have to be modified, depending on where it is positioned, as you're changing from pan/scan to letterbox mode when viewing on 4:3 TV.

1.Conversion to other display modes data is stored independently for buttons and for menu domain (probably even separately: in ifo for menu domain and in menu VOB for buttons, excluding CLUT stored in ifo).
2.There can be 8 group id's for different display modes:

0 normal
1 wide
2 letterbox
3 wide, letterbox
4 pan/scan
5 wide, panscan
6 letterbox, pan/scan
7 wide, letterbox, pan/scan

You got it! Display attributes and CLUT are stored in the IFO, whereas button info (button highlight info, color scheme table, button navigational commands) are stored in the navpack of a VOB.

The button group info (part of the button highlight info) is stored as 2 bytes: number of button groups and 3 possible button group types, each type having only normal/wide, letterbox, or pan/scan. For 4:3 menu, there is only one button group, so the group type (groupId) is 0. For 16:9 menu, there can be 2 or 3 button groups, so group 1 is often wide (groupId is 1), group 2 is either letterbox (groupId 2) or pan/scan (groupId 4). If there is a third button group, then group 3 is either letterbox or pan/scan.