video programming - where to start?

[sanlyn]

Already spent several hours last week and 2 hours this A.M. searching this forum and the 'net. All of which leads to this question:

Where does one start, from scratch, if one already has several years of Windows programming experience but has never programmed with graphics or video? It seems there are two areas one should start with: (a) characteristics of analog and digital video, and (2) programming languages. Somehow, I find no thread or 'net site that begins with the basics.

Amazon and other sites list literally hundreds (thousands?) of books, but none appear to tackle these subjects from the start.

My experience is mostly with Visual Studio, pre-. NET. Let's forget about.NET for the time being, I have no intention of getting into networked or 'net programming. My VB experience is extensive and includes methods for calling Windows library functions without the use of C++ (though I do know a bit about C++, if you are advanced enough with VB and system calls, you don't need C++ or Assembler to a high degree. You just have to know how the library API's work). On the other hand, I'm willing to get into VC++ if that's what it takes. Since 1982, I've taught myself 8 programming languages.

I've never been able to locate a basic guide to the nature of analog/video work. I bought two books, at a price, but they simply posed more questions than they answered.

Anyone willing to advise where to start from scratch, even with prior programming experience?

[usually_quiet]

I'm not learned in video programming, but I can offer a couple of hints based on tools I've downloaded from this site.

Many of the video programs here are actually GUI front-ends and script generators for some of the more powerful command-line programs, such as dvdauthor or avisynth. Learning those packages might enable you to do quite a lot in the same way.

As far as programming languages go, I've seen programs in the tools section of this site that the author said were written in C, C++, or Java, as well as a few languages I've never heard of, but not in VB, that I can recall. That doesn't mean there aren't any, just that I don't remember any that the author said he created using VB.

[TheFamilyMan]

Graphics and video aren't necessarily the same thing. I've written a few 3D graphics applications that use OpenGL and OpenGL does a very good job hiding all the low level video stuff. SGI's web site has LOTS of online technical documentation on graphics programming and also on some aspects of video and audio programming as well. Another good place to get your feet wet with low level video stuff would be with the virtualdub source code. I imagine all this comes down to discovering the right APIs: what they do, and how to use them (and obtain them). Good Luck!

[sanlyn]

Thanks, usually_quiet & FamilyMan, During the past couple days I've been getting into some your suggestions and will follow-up more -- well, after Santa comes and goes (pain in the neck. In the old days you said a prayer, thanked your friends and family, and turned in early). And as The FamilyMan suggested, I downloaded and unzipped source code for several simple filters, and looked into VirtualDub source.

As for VB, well -- obviously the biggest prob with VB is that a machine has to have the VB runtime. Not all that large, really, but many don't have it. As for Java, well, you need their 16MB runtime that makes VB's runtime look like a peanut. All the source code I've seen, btw, is in C++. But it has certainly given me some insight into the details of processing many types of digital video.

Guess it's time for me to dig up my old C++ books and the API references. Also, thanks for suggesting SGI. This project will do one of two things: (a) keep me out of trouble for a long, long, time, (b) get me INTO trouble for a long, long time.

[JohnnyMalaria]

I don't quite understand what you are trying to do.

For programming video files that have been imported or otherwise created and exist on your computer, the most common framework on Windows is DirectShow. It is very powerful, extensible and lightweight (i.e., little overhead therefore fast). But it has quite a steep learning curve. It is C++ based (though VB wrappers exist that don't provide the full power) and it uses a technology called COM (component object model). Like C++, COM is scary at first but once mastered, it is exceptionally powerful.

All the DirectShow stuff - including samples - is provided with the current Platform SDK.

DirectShow provides the means to read/write AVI files, capture from a camcorder, read/write MPEG, display video and whatever 3rd party plug-ins can do.

That's the programming bit.

As far as video goes, basically you are dealing with a series of bitmaps and, if you want to apply an effect, you create you own DirectShow plug-in (called a filter) and have DirectShow pass frames to your plug-in one at a time. There are examples in the SDK. Our software is entirely based on DirectShow as are many expensive editors.

If you can provide a bit more detail on what exactly you are trying to do, I'll be happy to try and help.

[sanlyn]

John, you're on a similar track. Ultimately I hope to get into a frame-by-frame cleanup of VHS tape captures (or, for that matter, cable tv video captures). Not that I intend to filter/clean/restore an entire movie this way, but some short sequences require massive fixes.

As it is, I've recently run into some DirectShow material, thanks to previous suggestions in this thread about reading software source code -- that's a pain in the neck, but it certainly leads to many particulars about all sorts of existing algorithms and support libs.

Basically the fact is that none of the commecial software or freeware I've found has specifically addressed some video restoral problems I've encountered. I figure I'll end up programming my own short routines to perform very specific restoral processing on short, very specific video clips.

I'm familiar with VB calls to C++, assembler, and COM routines, and I have a full Desaware library of VB enhancements plus other 3rd-party stuff -- true, you just about have to learn a lot of C++ to use VB with this stuff, but Desaware does help cut down on the C++ learning curve. All this has just come together over the past couple of days thru research paths suggested here.

As I said ealier, looks like I'm really asking for trouble. I guess it all depends on how valuable you think the results would be.

[zoobie]

Start by spending a few hours reading the glossary here @ VH
from A to Z

[jagabo]

start here:

http://www.amazon.com/Computer-Graphics-Principles-Practice-Programming/dp/0201848406/

Then get a good book on digital signal processing with emphasis on video.

[JohnnyMalaria]

I agree with jagabo.

From what you describe, you are wanting to create and implement data processing algorithms that work on 2D data.

Before jumping into the real video processing part, I would start with simple programs that work on bitmaps. e.g., a program that loads a single bitmap from disk that then processes it with your algorithm. Use this to optimize the algorithm (or at least get it to proof-of-concept) and then extend it to a plug-in (e.g., DirectShow) that can be used within video-specific software.

[jagabo]

Some usefule web sites:

http://fourcc.org/
http://www.poynton.com/
http://lipas.uwasa.fi/~f76998/video/
http://www.w6rz.net/

The blog archive at http://www.virtualdub.org/ has lots of musings on video processing issues. For example:

http://www.virtualdub.org/blog/pivot/entry.php?id=86
http://www.virtualdub.org/blog/pivot/entry.php?id=65

[ronnylov]

Creating short video restoral routines is exactly what you can do with avisynth scripts. Perhaps looking into how avisynth works and how to use existing avisynth plugins (or make your own avisynth plugins) is a good way to learn how to process video.

http://avisynth.org/mediawiki/Main_Page

[sanlyn]

ronnylov, I've been using Avisynth for some years now, but their plugins and those for VirtualDub fall short of some of my very specific needs for filtering really problematic videos. I'd be designing my own plugins or, for that matter, standalone processors.

I've found a lotta stuff since I started this post, and thanx to jagaboo for some more sources. I can use a lot of it with VB6, but it's apparent that for some really heavy-duty stuff I'll have to re-acquaint msyelf with VisualC for some of these projects. No free lunch, folks.

[vhelp]

I too have been researching this area. So I know what you must have been going through,
though my programming background is Pascal. Now I'm not sure about this, but in VB, I do
think they have a Scanline function (or equivalent) that accesses the bitmap data
directly. This has the advantage of greately speeding up the image processing if you want
to do it all in pure VB. Most image processing are either 24 bits and 32 bits, so you'll want
to know how your image manipulation routine handles those two common formats. Every
language has its unique way of dealing (programming) these types of areas criptic at first.

24 bits and 32 bits bitmap are the same, except that with the 32 bits, you have the ALPHA
channel for transparency -- like the watermark Logo's you see on TV -- but the structure
will be different, algorithmicly.

I also wrote a short article based on a Median algorithm with a few demos of the work that
this first-step in filter developing produced. Not the greatest stuff mind you, but is a start
in the direction I am interested in -- Video Restoration.

--> Image Processing: Temporal / Spacial Median filter -- December 2007, by vhelp

I'm hoping to get my first AVIsynth filter plugin developed, but its not been easy, for me, that is.

Anyways..

So in addition to the other's posted..

The things that you really want to get into is Bitmap and AVI structures. Learning how they
are put together will help you to understand "how day do dat". But, these two elements are
key areas to learn.

Also, look into terms like Spacial and Temporal, and how they are applied to the bitmap.

And there is also, the color spaces, RGB; YUV; YUY2/YV12 are some of the most common
color spaces and formats to be researching on.

Most plugin/filters are used in AVIsynth are in YUV color space, and either use the common
YUY2 or YV12 formats. Coding/developing plugins for AVIsynth require c/c++ but is mostly
platform specific -- Microsoft or something.. I forget.. so, gcc; ming; etc., forget it. won't
work.

VirtualDub can have plugin/filters developed too, but also require c/c++ as above.

-vhelp 4505

[sanlyn]

Thanks, vhelp, for those observations. Yep, I'm certain that C++ is the way to go, and during the past few weeks I've gotten right back into Visual Studio 6, for both VB and C++. Fortunately I find VS6 with its service packs is fully capable for today's systems, including Vista. I also recently found that armies of programmers have given up on .NET stuff and newer; they are going back to VS6 routines, which have a few more years of debugging that gives a big boost in reliability.

Another note: I don't know if you are familiar with Desaware add-ons. They make it possible about 90% of the time to use VB6 as if it were C++. You can't do it at all times, but when it comes to setting up a complex GUI for calling lower-level routines, VB6 and Desaware can save weeks (even months) of work. At some point, though, you do have to get down to lower level routines or at least understand how Windows uses ready-made support files coded in C++ or Assembler.

If I need to do any processing in YUY2/YV12, I see that AviSynth's existing plugins do pretty much everything I would want to do in AviSynth. Because my favorite decoders and VirtualDub work in RGB, I'm concentrating on that. This allows me to avoid colorspace conversions as much as possible. True, the final DVD output colorspace won't be RGB, but most of the work along the way is far more predictable if you stick with one colorspace. And as things are, a PC monitor is going to use RGB anyway.