[xyzxyz]

Hi,

I purchased Downloader Pro 1.6.2. I'm seeing about 1 out of 1000 or 2000 images corrupted when I download them with Downloader Pro. I shoot a lot of photos, so actually it is happening quite frequently. When I download the same image again, it has been okay the second time everytime... but I don't feel comfortable with the image corruption. I may not catch every corruption.

I've seen corrupted images using both my Sandisk 1GB compact flash card and my Hitachi 4GB micro drive. Those are the only two cards that I have. I reformat after every use.

I'm running Windows 2000 with Service Pack 4 with a Sandisk compact flash reader plugged into a PCI USB2 adapter card.

Do you have any suggestions?

Thanks!

[Chris Breeze]

It sounds like it may be a hardware or driver issue. If the images are OK when they are downloaded a second time then the originals on the CF card must be OK. I suspect the problem may be related to your card reader or its drivers. It's worth checking SanDisk's website to see if they have a newer driver for the card reader. The other possibility is the PCI USB2 card.

I don't use Win2000 any more, but when I did use it I remember having huge problems with drivers for video cards, card readers, USB2 adaptors etc. I haven't had any such problems since I switched to WinXP Pro. I download well over 100,000 images each year and have never had a problem with corrupted images.

[xyzxyz]

Actually I used a CRC utility and the corruption is happening much more often than I thought. It's probably just not easily noticeable unless the certain bits are corrupted, resulting in a very obvious change in the JPEG.

Today I shot about 4 GB of images in JPG + RAW. There's 20 or so JPG's that are corrupted according to the CRC utility. Strangely not a single RAW file was corrupted. This leads me to suspect that the problem that I am experiencing is a bug in Downloader because the card reader driver should not care what kind of files are being accessed.

[xyzxyz]

I looked at the issue in more detail this morning. Actually it's 9 JPG's of the 688 JPG and CR2 (RAW) files that I shot yesterday that are not exactly identical between the two times that I downloaded them using Downloader Pro.

I looked at the differences with a binary diff tool. 8 of the 9 discrepancies occur in the region (roughly) of byte number 4500 (in hex) / 17664 (in decimal). These 8 files are vertical shots. (I have Downloader Pro set to automatically rotate JPG's). Only a small percentage of my vertical shots from yesterday show these byte discrepencies.

Two bytes are changed.
Usually the bytes that are different were changed to 00 the second time that I downloaded the images, though not always.

This appears to be in the IPTC data region, because I see my name nearby. (In Downloader Pro, I have it set to add my name to the copyright field. That is the only IPTC data that I am explicitly adding in Downloader Pro.)

Can you think of a reason why the files would not be byte-for-byte identical? I didn't change any settings between the downloads.

In the remaining file that shows a discrepancy, the byte corruption occurs much later (I assume in the image data, though I don't see an obvious artifacts in the photo). It is a horizontal shot. It seems possible that there are in fact two corruption problems, one very likely in Downloader Pro, and the second in an unknown component.

[xyzxyz]

I spent more time looking at the issue. I've tried downloading the photos many times, and this is what I have discovered:

The subset of vertical-oriented images which differ keep changing. There are some images which seem particularly prone to changing when downloaded.

To test, I also tried downloading on a different computer and totally different card reader (Windows XP this time with a Delkin card reader), and I see the same issue... some subset of the vertically oriented files keep mutating every time they are re-downloaded. The set of vertical images that were affected seemed to be different than on the first computer, with some overlap.

Do you know what this is happening? The images with the byte mutations look okay, even though 2 bytes are mutating.

I also installed newer USB2 drivers on my Windows 2000 computer. I hope the problem that I have with images visibly being corrupted is fixed.

[Chris Breeze]

If you select the auto-rotate option in Downloader Pro it will losslessly rotate vertical JPEGs as they are downloaded. This will result in the downloaded files being having a different CRC from the originals on the CF card. This doesn't mean they have been corrupted though. The raw files are not modified (unless you are adding IPTC data) and so you would expect the CRC of the originals and the downloaded files to be identical.

[xyzxyz]

Please re-read my posts. I'm not comparing the downloaded files with the files on the media!
I'm comparing downloaded files with the same downloaded files, that were downloaded previously.
Everytime I re-download the files, a subset of the downloaded files are slightly different than the files that were downloaded before.

It happens to some of the rotated files, and the difference is at the byte number that I indicated before.

[Gary_Berg]

I'm wondering if when DL Pro rotates the file and writes EXIF/IPTC information if something in the header is updated with date/time?

[xyzxyz]

It doesn't always change a particular file. It only sometimes changes a particular file. But it always changes some of the files... usually 8 - 20 files. A different subset of the rotated images is changed each time... in the area of the IPTC info.

From a programming perspective, it might be a buffer that's not being initialized, and thus non-deterministic data is ending up in the file.

[Chris Breeze]

Are the files actually corrupted (i.e. are unreadable) or are they just different by a couple of bytes? Some of the data stored in the JPEGs is WORD aligned and may be padded with possibly uninitialised bytes to make them the right length.

[xyzxyz]

Hi Chris,

I haven't seen any visibly corrupted images related to this issue of the rotated image files changing with each download. (I have seen visibly corrupted images, but as I've said, it is most likely an unrelated problem (possibly with my CompactFlash reader/drivers/OS).)

Can you change the code so that all the buffers are initialized or padded with zeros? It's not desirable at all that rotated files change with every download. :-)

Thanks,
Peter

[Chris Breeze]

There's not much I can do if this is caused by the libraries used to rotate the JPEGs. I'm not sure that it matters anyway provided the data is valid.

[xyzxyz]

> There's not much I can do if this is caused by the libraries used to rotate the JPEGs.

I'm not sure what you mean. If the code is open source, then either you or someone else can fix it. If you are using a commercial library, then you can file a bug report with the vendor, and they can fix it.

I understand it may not be a very high priority bug to you, but to say there's not much you can do isn't true.

[it's a lifestyle thing]

If it's any help this is my workflow - and to date I've had no problems.

1) Manually download files of cards to HDSC - and I compare files sizes.
2) DL Pro - renames and collates all the seperate files from multiple cameras and folders into one location. It also adds exif - so I obviously wouldn't compare file sizes any more, as they would have changed.
3) Now my auto process routine, open the DL Pro files, does my 'exclusive file processing routines' then saves the files out to a new folder.
4) Use Breeze Browser Pro to select, and grade.
5) BackUp to HDSC and DVD

Step 3/4 pretty much guarantee no curruption - without me being aware. Only at this stage would I put the memory cards back into the camera bag as ok to format.

I only worry about image corruption, and don't in a workflow try for identical file sizes.

Regards,
Kev

[Clayton]

I seem to be having the same problem. Using 1.6.2 version.
Windows xp pro and Scandisk imagemate 8in1 reader. I have 2-Nikon D70 - one D2H and 2 - D2x's. I am using lexar 1gb cards, one hitachi drive and an IBM microdrive. I get one or two corrupt image everytime I download.

I will try to do some more testing - I just wanted to say someone else is also having problems along this line.

clayton

[Chris Breeze]

The problem may be due to the card reader. I don't have a great deal of confidence in cheap 8-in-1 card readers (I have an unbranded one which was frankly a waste of money). I use a SanDisk USB 2.0 card reader on my desktop and have found it fast and reliable. On my laptop I usually use a Delkin CardBus 32 card reader.