Handling 1 million jpegs in a windows directory

[deanbrown3d]

Hello all,

Great to see such a great site - this is my first question

I have a security camera taking jpeg shots every second and storing on a 3/4 TB HD (in a DLink data/FTP server). After a few days, when I have some number of 100,000's of pictures, I cannot copy/paste the directory - I cannot do anything, even look at the directory in windows explorer. I think its a resource issue with Windows trying to load image file headers into memory. How can I handle such data? I'd like to either:

1. Select all files within a specific date/time (for example 15 minutes' worth, if an alarm was tripped and I want to see what was happening) and copy to another directory for analysis.

or

2. Convert the jpegs into an mpeg at periodic intervals (say every hour) so that the file count is limited to a few thousand rather than millions. Is there a program that can handle the large file count?

Before doing any conversion, the first hurdle (to move a subset to another directory) has to be solved, since the files are stored on the server and not on a PC's hard drive.

Thanks for any tips and sorry if this has been covered to death - I did a search here and on google groups and could't find anything specific to large file counts.

Thanks!

Dean

[Nelson37]

Haven't seen this before, but sounds similar to hitting the nested directory limit, problem could be severe. Don't recall the limit on files within a single directory, it could just be taking forever to generate the thumbnails or even just the detail view.

First, investigate storing in different directory each day, or rename the current used one and create a new one. This problem should have been foreseen and eliminated from the beginning.

Hopefully the filenames offer some division, ie 091807520pm.jpg or something similar. You can run the DIR/P command to get an idea of the filenames. Try going to the command prompt and do something like "xcopy 091807*.jpg c:\newdir1" with "newdir1" representing a new directory you have created. Then repeat for different days until the content is small enough to deal with.

You could create a batch script in TMPGENC or something similar to MPG the older directories, or seperate by filenames, you may have to load into VDUB and frameserve. This process is fairly straightforward but may interfere with the capture process. Most such equipment can generate MPG files or other compressed format directly, individual JPG's is unusual. Is there some reason the JPG is being used?

[deanbrown3d]

Nelson -

Thanks for the reply. The files are stored in name format with the date and time stamp to the nearest 1/1000 sec, so they are in order by name and by date.

The camera (infact all security cameras I have looked at) can only transfer jpegs when set on timer mode. You can get the camera to send an mpeg when it is tripped by an alarm or motion detection, but in my case I want it recording all the time, every second. Its going to a concealed FTP server.

I could write a little program that renames the directory every few hours from a PC, plus even do the mpeg conversion, but that means I would need a PC on all the time. Simpler would be to just let it accummulate millions of files over the months, but of course I need to be able to access them if the time comes.

[Soopafresh]

http://en.wikipedia.org/wiki/NTFS for some file limit specifics.


Possibly a useful utility - http://www.i386.info/datedir.htm

Another util you shouldn't be without - http://www.kessels.com/JkDefrag/

Use Avisynth and a script to make videos on a timed basis. You need to download the imagesequence.dll plugin to do this:


@echo. >>enc.avs
@echo CoronaSequence("*.jpg", sort=1) >>enc.avs
@echo assumefps(23.976) >>enc.avs
@echo bicubicresize(720,400) >>enc.avs
@echo converttoyv12() >>enc.avs



x264.exe --pass 1 --bitrate 800 --stats ".stats" --ref 1 --no-b-adapt --progress --keyint 250 --bframes 1 --qpmin 8 --qpmax 51 --no-psnr --filter 0,0 --subme 1 --direct auto --vbv-maxrate 25000 --me dia --no-ssim --level 30 --no-cabac --analyse none --threads auto --thread-input --output NUL "enc.avs"

x264.exe --pass 2 --bitrate 800 --stats ".stats" --progress --keyint 250 --bframes 1 --qpmin 8 --qpmax 51 --no-psnr --trellis 1 --ref 5 --filter 0,0 --direct auto --vbv-maxrate 25000 --me umh --no-ssim --level 30 --no-cabac --b-rdo --analyse p8x8,b8x8,i4x4,p4x4 --threads auto --thread-input --output "video.mp4" "enc.avs"

[deanbrown3d]

Thanks Soopafresh

A snipet for those interested:

Max number of files 4,294,967,295 (2^32-1)

It does not specifiy if this is for one directory, though I am reassured somewhat (since I am only 1.2 gigaseconds old).

[ChrissyBoy]

If your jpegs have Exif data then AmoK Exif Sorter is your friend....

Sort your massive folder into new folders order by year/month/day/hour for example...

[AlanHK]

I have a security camera taking jpeg shots every second and storing on a 3/4 TB HD (in a DLink data/FTP server). After a few days, when I have some number of 100,000's of pictures, I cannot copy/paste the directory - I cannot do anything, even look at the directory in windows explorer. I think its a resource issue with Windows trying to load image file headers into memory. How can I handle such data?

Open "My Computer"
Tools/Folder Options
General tab:
Turn OFF active desktop
Turn OFF "enable web content"


And see http://www.annoyances.org/exec/show/article03-203
for how to set your view (to "details", instead of "icons") and disable the Thumbnail cache.

[deanbrown3d]

If your jpegs have Exif data then AmoK Exif Sorter is your friend....

Sort your massive folder into new folders order by year/month/day/hour for example...

Thanks - I'll look into it. Is there some obvious way to tell if there is exif data in a jpeg?

[deanbrown3d]

...And see http://www.annoyances.org/exec/show/article03-203
for how to set your view (to "details", instead of "icons") and disable the Thumbnail cache.

Hi Alan - does "disable the thumbnail cache" mean the same thing as just showing the details instead of icons?

[AlanHK]

...And see http://www.annoyances.org/exec/show/article03-203
for how to set your view (to "details", instead of "icons") and disable the Thumbnail cache.

Hi Alan - does "disable the thumbnail cache" mean the same thing as just showing the details instead of icons?

You should read up on it at the Annoyances site; but I think the thumbnail cache stores all the little images used for the icon for each image. Creating this for thousands of files is what's locking your computer up. If you can just display "details", i.e., a list of filenames, it should be much faster.

[ChrissyBoy]

If your jpegs have Exif data then AmoK Exif Sorter is your friend....

Sort your massive folder into new folders order by year/month/day/hour for example...

Thanks - I'll look into it. Is there some obvious way to tell if there is exif data in a jpeg?

Well if you can - just hover your mouse over an image and you will get a popup with camera model/date taken etc... this is the exif data.

Or do a right click-> on an image:

[deanbrown3d]

If you can just display "details", i.e., a list of filenames, it should be much faster.

Yes, of course I have turned that off. Will do the other recommendations tonight.

-Dean

[deanbrown3d]

Or do a right click-> on an image:

I get this dialog when I right-click and select Properties:

[ChrissyBoy]

Choose the Summary tab.

[deanbrown3d]

Choose the Summary tab.

Dammit I'm sure I looked at that! It was on the advance button of the summary tab. I assume this picture means I don't have exif data: