If you want to synthesize NTSC (which has absolutely nothing to do with VHS) yes, you can do it in software but the result is a digital stream anyway. You still have to convert it back to analog using a fast DAC so it becomes rather pointless when there are cheap hardware solutions that can directly convert RGB or YUV costing only a few $$. You would be emulating an IC like the AD720 series in software. If you really want to try it, the first place to search is for VHDL models of the AD72x series as these will have most of the math in them already.
Sounds like an interesting project itself.
Where would you download these VHDL files? And what software would you use to run them? In my quick look for VHDL emulators, I found some software that looks professional, like it would be very expensive software. Is there any hobbyist level software for running VHDL code?