Help encoding 360 x 360 videos (spherical movies)

[sapojow]

Hello everyone,

I have been working with a technology that allows me to create videos like this one www.imervista.com.br/demo/rio and www.imervista.com.br/demo/wg7br (full screen) (once the movie starts, click and drag the image)

The movies are captured by our camera and delivered in .AVI format (m-jpeg codec). The source dimensions are 2400x1200, and the bit rate is 29000kbps

Up to this point is fine. video quality is great. but then we have to use final cut to edit the movie and after that we couldn't find a
formula which has a decent performance/quality compared to the initial .avi. The final format I need is either h264 (mov), .flv, or .f4v

Usually we use two methods:
* first method: (recomended by the manufacture) after editing we export from final cut in avi and then encode again (avi) using virtual dub, resulting to another .avi. I then encode them using quicktime pro, compressor using the following parameters:
Video: h264 (mov)
size: 1200 x 600
bit rate: 5000 kbps
frame rate: 29.97
key frame: every 30 frames
audio: AAC, 128 kbps

result sample video: www.imervista.com.br/demo/rio

* second method: export from final cut straight to .mov using the same parameters above

sample result: www.imervista.com.br/demo/wg7br

The second method give me better results, however, at 5000 kbps is still to slow for web streaming (and still not as great as the avi original movie)

I am looking for a solution in which I could use flv or f4v and that would still carry good picture quality and would have a faster streaming...

anybody could help me figure this out?

note: the delivery formats must be flv, mov, or f4v. The audio must be mp3 or AAC (44100 Khz)

I would appreciate any responses.

Cheers,

Alex

[minidv2dvd]

personally that sounds like a long and winding road. any chance if you have to work on a mac you can get premiere pro cs4? edit in it and encode direct to flv.

sorenson sqeeze pro would be my second choice for an encoder to produce flv. it includes the on6 fast start option.

that first version is useless. you have to download the entire video before it starts. you need an h264 encoder that puts the mov atom at the beginning so the video starts right away. try mainconcept or x264, but you would probably have to use the mp4 container. i gave up after 5 minutes of empty black screen.

second one works but looks a bit washed out. i was getting an average 5mbps from your server which is pretty impressive. the video pauses at odd times while the audio continues for not apparent reason. it will probably overwhelm most systems. it was using about 33% of the cpu cycles on a quad core 3GHZ system to play. add to that the 5mbps download speed needed to play it and you are going to frustrate a lot of users.

pretty cool though. i'm not familiar with the skyline. rio?

[sapojow]

minidv2dvd, thank you for your straight answers. I will look into mainconcept to see what I can get out of it. For now, using premiere pro is not an option... so I will have to find a solution using final cut pro.

Can you recomend any online course or tutorial that can start me up in how encoders work and exactly what they do? For example if I want to become a codec expert... where should I start?

you are right about the skyline... it was made in copacabana, Rio de Janeiro...

thanks a lot for the reply... I will try your recomendations and see what I will get out of it...

Cheers,

Alex

[ricardouk]

5000 bitrate? web streaming?

With dimensions as 1200x600 you need to pump up the bitrate...when i stream videos in MP4(x264+aac) i only use a maximum width of 640 and since aac compresses better than mp3 i use a bitrate of 64, sample rate 22050 stereo, i dont think you need to use 128 for those videos.

I recommend xvid4psp, select mp4 as format, select x264q16 medium profile for video profile and then do a 2 pass encode.

for audio i select the aac-lc 64 CBR profile

Hope it helps

[minidv2dvd]

in final cut pro can you import the source into apple prores 422? if so keep it in that lossless codec to prevent generational losses until you render to flv or mov.

i think if you reduce the size to 720x360 you can get away with a bitrate of around 2000kbps and maintain good streaming quality.

what company made the cam? i like the effect of the 360 lens, but they need to work on the blurring at the convergence of the fisheyes a bit. you have a little editing to do in places to remove the cameraman....

[jagabo]

Note that all high compression codecs get a large part of their compression by not including parts of the picture that don't change from frame to frame. So a video like your first one (from a car driving down the street) will require lots of bitrate because of all the motion. The second video (a stationary camera on top of a building) will compress better because most of the frame doesn't change from frame to frame.

5000 kbps with a 1200x600 30 fps video will be pushing it if you have lots of motion. Note also that video static noise is the equivalent of motion when encoding video. Ie, the noise is different in every frame so the encoder will have to waste bitrate on it.

[sapojow]

5000 bitrate? web streaming?

With dimensions as 1200x600 you need to pump up the bitrate...when i stream videos in MP4(x264+aac) i only use a maximum width of 640 and since aac compresses better than mp3 i use a bitrate of 64, sample rate 22050 stereo, i dont think you need to use 128 for those videos.

I recommend xvid4psp, select mp4 as format, select x264q16 medium profile for video profile and then do a 2 pass encode.

for audio i select the aac-lc 64 CBR profile

Hope it helps

The issue with reducing the dimensions of the player is the following: In a normal flash player, you would have a proportional dimentional "stretch" of the image, to allow it to see in full screen. on this case, in order to create a spherical enviroment, the player gets the image and it stretch it into a sphere. Now think of this, when you are seeing the player on the screen at a 640 x 230, you are in fact, seeing a "portion" of the sphere from inside of it. which means that I have to have more image area "around" the player, se the user can the most of it with the minimum of distortion. So, I need an enough of large work area to work with. around 1024 x 512 is the minimum I could go in order to have a decent watchable window area.

I will try it out using your specs on the audio part, but I am assuming that will reduce very little of the total file, since the audio part is just a little part of the whole wrap.

cheers,

Alex

[sapojow]

in final cut pro can you import the source into apple prores 422? if so keep it in that lossless codec to prevent generational losses until you render to flv or mov.

i think if you reduce the size to 720x360 you can get away with a bitrate of around 2000kbps and maintain good streaming quality.

what company made the cam? i like the effect of the 360 lens, but they need to work on the blurring at the convergence of the fisheyes a bit. you have a little editing to do in places to remove the cameraman....

Check the answer I gave ricardouk about the dimension size. We have different sources for cameras...but the samples that I have showed it to you were acquired using a camera from www.immersivemedia.com, they have their own proprietary process, which works pretty well. I am trying to R & D about other process, and maybe find out a new workflow to stream HD (the current cameras are SD quality)

[sapojow]

Note that all high compression codecs get a large part of their compression by not including parts of the picture that don't change from frame to frame. So a video like your first one (from a car driving down the street) will require lots of bitrate because of all the motion. The second video (a stationary camera on top of a building) will compress better because most of the frame doesn't change from frame to frame.

5000 kbps with a 1200x600 30 fps video will be pushing it if you have lots of motion. Note also that video static noise is the equivalent of motion when encoding video. Ie, the noise is different in every frame so the encoder will have to waste bitrate on it.

you are absolutely right... my goal is to find a way to stream 30fps (ideally it has to be 30fps for media projects) with a lower bit rate... the bit rate is what is killing the user experience... like minidv2dvd said. It will frustrate a lot of users...

[sapojow]

My goal is to get the avi file transcoded to flash or F4V, with a h264 encode, with a quality close enough of the original avi file. and If I can get that with a bit rate lower than 2000 kbps, I would have succeed. am I dreaming to high?

thanks for all your guys comments

Cheers,

Alex

[jagabo]

You will simply have to have different sizes for different users.

[minidv2dvd]

have you got a source sample we could experiment with?

[jagabo]

Can you live with 24 fps? That may reduce your bitrate requirement by about 20 percent.

[sapojow]

Can you live with 24 fps? That may reduce your bitrate requirement by about 20 percent.

I will try We are dealing with the entertainment industry... they are tough about those fps... but I will experiment it

[sapojow]

have you got a source sample we could experiment with?

I will extract a virgin one tomorrow.... one minute of video = around 500 mb

[sapojow]

Hey Everyone,

Sorry it took me so long to respond... trying to get a new immersive job done. So I encoded 9 versions of a 3 min video using encoding.com. www.imervista.com.br/groundzero through the menu of the player, you can watch all the versions. I noticed that 1200x600 is the size that worked the best so far, and I got mix feelings about the other ones. I lowered the bit rate to 1600 and quality has suffered. Could you guys tell me what are your experience watching them? Here is what I got so far (I am on a 6Mbps internet cable connection:

Video go or no go/reason loading speed

original mov ------------- go --------------------------------------- 2m 10s
01------------------------- no go/video and audio skipping------------- -----
02-------------------------- go -------------------------------------- 44s
03--------------------------no go/video and audio skipping ------------ -----
04-------------------------- go --------------------------------------- 40s
05-------------------------- go --------------------------------------- 48s
06-------------------------- go --------------------------------------- 44s
07-------------------------- go --------------------------------------- 43s
08-------------------------- no go/no video ---------------------------- -----
09-------------------------- go -------------------------------------- 45s

Thanks a lot!

Alex

[jagabo]

Another issue you have is the non-linear projection. That eliminates most of the motion vector techniques used in h.264 encoding.

[minidv2dvd]

is the file you call final1.mov the source video straight from the cam? it's already in acvhd mpeg-4 with a variable bitrate of only around 4000 kbps, so any further re-encoding to lower bitrate mpeg-4/flv is going to look bad.

[sapojow]

is the file you call final1.mov the source video straight from the cam? it's already in acvhd mpeg-4 with a variable bitrate of only around 4000 kbps, so any further re-encoding to lower bitrate mpeg-4/flv is going to look bad.

no, it is not the original ( the original has close to one 1GB, mpeg, about 40000 kbps). What i called original was just the first one I did... sorry if I didn't explain that before

[minidv2dvd]

do all the videos look ok when viewed in a "normal" player, i.e. no macroblocking or other artifacts? i downloaded the final1.mov and it looks fine in vlc as a weird shaped video. no visible defects. i'm wondering if the defects are introduced by the 360 degree player, as i would reject all those test videos as not good enough to use.

did you ever get around to making the test source file?

I will extract a virgin one tomorrow.... one minute of video = around 500 mb

if so, have link to the file?

[jagabo]

do all the videos look ok when viewed in a "normal" player, i.e. no macroblocking or other artifacts?

A 2400x1200 frame, 30 fps, at 40,000 kbps MJPEG is going to be full of macroblock artifacts.

[minidv2dvd]

do all the videos look ok when viewed in a "normal" player, i.e. no macroblocking or other artifacts?

A 2400x1200 frame, 30 fps, at 40,000 kbps MJPEG is going to be full of macroblock artifacts.

are you referring to the original source? he lists it as mpeg not mjpeg.

the original has close to one 1GB, mpeg, about 40000 kbps

[jagabo]

do all the videos look ok when viewed in a "normal" player, i.e. no macroblocking or other artifacts?

A 2400x1200 frame, 30 fps, at 40,000 kbps MJPEG is going to be full of macroblock artifacts.

are you referring to the original source? he lists it as mpeg not mjpeg.

the original has close to one 1GB, mpeg, about 40000 kbps

His last post listed it as MPEG but his first post listed it as MJPEG AVI.

[sapojow]

do all the videos look ok when viewed in a "normal" player, i.e. no macroblocking or other artifacts?

A 2400x1200 frame, 30 fps, at 40,000 kbps MJPEG is going to be full of macroblock artifacts.

are you referring to the original source? he lists it as mpeg not mjpeg.

the original has close to one 1GB, mpeg, about 40000 kbps

Sorry I misspelled it, It is mjpeg (avi containter) not mpeg

[jagabo]

You might try using a deblocking filter on your MJPEG source. Deblock before resizing and compressing.

Oh, and when you resize different algorithms can make the video easier or harder to compress. Bilinear or Precise Bilinear will be easier to compress (not quite as sharp, less high frequency noise), Bicubic or Precise Bicubic will be harder (sharper, more high frequency noise). Basically, the sharper the resizing algorithm the higher a bitrate you will need.

Noise reduction, both spacial and temporal, can reduce the bitrate requirement.

Here's a short sample after filtering with Neat Video in VirtualDub and encoding with x264 at about half the bitrate:

http://www.mediafire.com/?sharekey=49c752f9633c890caf924764f9977b1dff48a68dab7e8639c95965eaa7bc68bc

This isn't optimal because I started with your already h.264 file. You would probably get much less posterization and overall better results starting with your original MJPEG AVI.

[sapojow]

do all the videos look ok when viewed in a "normal" player, i.e. no macroblocking or other artifacts? i downloaded the final1.mov and it looks fine in vlc as a weird shaped video. no visible defects. i'm wondering if the defects are introduced by the 360 degree player, as i would reject all those test videos as not good enough to use.

did you ever get around to making the test source file?

if so, have link to the file?

The videos will look good in a regular player, however, what you see in a regular player would be a "flat" image showing all the cameras at the same time. It looks like a "panoramic image", but with also the images of the sky (top camera) and ground (lower cameras). The addition of the top and bottom images make a "spherical image"

So in order for you to be able visualize a image inside of the video frame, the player gets that spherical image and wraps in the obviously shape of a sphere.

Now try to imagine yourself inside of the sphere. The bigger the sphere easier is for you to see things without distortion, because in your view filed you can see all the "edges" of the sphere. Just like our planet. But in the real world, you don't see the edges of the planet, therefore you can't realize the distortions (have you seen those pictures of the earth seen from space? you see the whole planet and the countries seem to be round, just like the sphere which they are in. Which is actually true, but you don't perceive that way, because of the size, gravity, rotation, etc, etc)

So in order to keep everything looking squared and not curved/distorted, the player try to correct this by stretching the image. that causes pixelation/macro artifacts to start to be seen) That's why in order for this technology to come to life in its real potential, the image has to be large enough in size, and/or in really high definition, so even if we "stretch" the image we wouldn't see the artifacts. That is currently done offline on the computers, but for obviously reasons still hard to do over the internet.

About the sample, I couldn't get my technician to upload the right one (I want to upload one with a lot movement) . The source videos are guarded in Brazil, in a proprietary format before they can be decrypted to MJPEG avi. They take a long time to be decrypted... even in very powerful computers. Anyway... here is what I got so far... this is data that comes straight from the camera, there is no filters applied, neither the parallax (the blind spots between the lenses, that show when you get too close of the camera) is adjusted. http://www.imervista.com.br/tools/test/wg7test1.avi

Thanks a lot!

Alex

[sapojow]

You might try using a deblocking filter on your MJPEG source. Deblock before resizing and compressing.

Oh, and when you resize different algorithms can make the video easier or harder to compress. Bilinear or Precise Bilinear will be easier to compress (not quite as sharp, less high frequency noise), Bicubic or Precise Bicubic will be harder (sharper, more high frequency noise). Basically, the sharper the resizing algorithm the higher a bitrate you will need.

Noise reduction, both spacial and temporal, can reduce the bitrate requirement.

Here's a short sample after filtering with Neat Video in VirtualDub and encoding with x264 at about half the bitrate:

http://www.mediafire.com/?sharekey=49c752f9633c890caf924764f9977b1dff48a68dab7e8639c95965eaa7bc68bc

This isn't optimal because I started with your already h.264 file. You would probably get much less posterization and overall better results starting with your original MJPEG AVI.

thank you for the sample. The image looks really good, but the playback is choppy. Here is a sample of the original source http://www.imervista.com.br/tools/test/wg7test1.avi It is not the ideal test because there isn't a lot of movement, but I will upload another one as soon as I get my technician free to do it.

Just to let you know the player that I use can be found here www.lucid.it

Cheers,

Alex

[sapojow]

just to let you know... the player that I use can be found here http://www.lucid.it

[jagabo]

Here's a short sample after filtering with Neat Video in VirtualDub...

thank you for the sample. The image looks really good, but the playback is choppy.

Hmmm. It plays smoothly here. Maybe you're having problems because I encoded in VirtualDub (to use Neat Video) as AVI, then remuxed the h.264 stream into an MP4 container. The x264vfw encoder produces slightly odd h.264 streams.

Here is a sample of the original source http://www.imervista.com.br/tools/test/wg7test1.avi It is not the ideal test because there isn't a lot of movement

Yes, that one will encode down to very low bitrates without problem. Here it is at 1200x600, x264vfw in AVI, about 800 kbps:

x264ts5.avi

I used the Lanczos Resize, MSU Deblocking and Temporal Smoother filters.

[poisondeathray]

Hmmm. It plays smoothly here. Maybe you're having problems because I encoded in VirtualDub (to use Neat Video) as AVI, then remuxed the h.264 stream into an MP4 container. The x264vfw encoder produces slightly odd h.264 streams.

That's not the cause in this case. It plays smoothly in "normal" players. When you run it through the Lucid flash player through the xml/html embedded player offline = jerky city. I tested using an older single core computer to simulate "worst case" scenario for possible end users. Some streams that didn't play well on the single core obviously played better on a faster rigs.

That modded flash application is nerfed. It doesn't play well with all h.264 settings. Adobe Flash coding is bad enough, but with the extra overhead imposed this is very bad. I encoded several streams and the same video plays fine in other flash player implemenations. (Also the visual quality is dramatically worsened with this player compared to other flash players, everything becomes less crisp, I suppose it's from the warping/wrapping technology?)

This means you have to use lower quality & lower compression settings in order to make it play smoothly (worse compression or worse quality for the same bitrate). You basically can't use AVC High (no I8x8 or adaptive DCT), 1 reference frame max (sometimes I was able to use 2), no b-pyramid, max 1-b frame. For anyone that thinks it's a x264 compatibility issue ; it is not. I tested some streams using Adobe Media Encoder for .f4v streams, and any "higher" quality settings would cause jerkiness. I tested some of the successful streams from your website, and the ones that "play nicely" for the single core were no b-frame, 1 ref frame, avc main. From my quick testing, you're basically stuck with using low quality settings, and compensating by using higher bitrates (which then raises the issue of bandwidth as the bottleneck, because I only did offline testing)

Going the preprocessing route with filtering is a good idea as jagabo suggested, but wow this player makes videos look very bad; and there's not much leeway you can do in terms of optimizing encoding settings. I can't think of any decent solutions here .

PS. there's some errors in the source video, there's only about 17sec of valid video frames (more for audio), but it's enough for these testing purposes

I think this technology is great! I love this 360 degree concept. It's just that adobe has to pick up the pace and fix their flash coding. Hopefully by Flash11 things will be better