Image recovery: Recovering fragmented image files from flash memory cards

[Tonny Iversen]

Hello. I'm trying to figure out if it exists any software that might be able to recover fragmented images from flash memory cards / digital cameras if the card was quick-formatted (i.e. if the card has a zeroed out FAT), or some details about the jpeg algorithm to figure out how difficult development of such would be.

My main interest is for .jpg/.jpeg files.

In the situation when the FAT contains only zeroes, recovery of non-fragmented image files seems to be easy for many (or maybe most?/all?) file / image /data recovery software products, but I haven't been able to find any that can recover anything more than the first fragment of the file if the file is fragmented.

I don't know the specifics of jpg encoding, but maybe someone here knows if the different parts of the image part of the jpg files are numbered in some way, or contains some patterns/fields that can be used to connect later fragments together with the first fragment? If not, maybe something like FFT/Fouier transformations/frequency analysis or something similar/else could be used to do a qualified guess about which fragments belong to each other and maybe present the user with some alternatives?

I guess that if image files are fragmented on a memory card, the chance is pretty high that the files only has two fragments. At least this is my guess if the card was new or formatted before use, then used to take pictures and erase some in between, and then quick-formatted afterwards. And if the directory entries are intact and most files are defragmented, I guess it could be possible also to do an educated guess about which clusters are fragments belonging to other files. Also, comparing the starting clusters from all files could indicate if the file is actually fragmented, if the difference between the starting cluster of file n and n+1 is less than the number of clusters necessary to store the file. And if the card was new, zero-filled or filled with a specific pattern before use, it should be obvious if a specific cluster was in use or not before the quick-formatting. (Maybe a full formatting fills the cards with a specific pattern? I haven't tried...)

Does anyone know of any software doing something like this or is anyone considering making something like this? I'm to pressed for time to consider developing something like this right now, but if someone else wants to look into this, I'd be more than happy to be involved in discussions around the development if you want (though not very much since I don't have much time available). And if someone could comment on these issues, it would be great if I or someone else should consider developing something like this.

Best regards
Tonny Iversen (***@online.no)

[Hans-Peter Diettrich]

Post by Tonny Iversen
In the situation when the FAT contains only zeroes, recovery of
non-fragmented image files seems to be easy for many (or maybe
most?/all?) file / image /data recovery software products, but I
haven't been able to find any that can recover anything more than the
first fragment of the file if the file is fragmented.

First you'll have to find out the cluster size, then the directories,
which give you the starting cluster, name and size of all the files.

With fragmented files you'll have to combine every starting cluster with
every other (not yet assigned) cluster on the medium, a very time
consuming process. Also consider erased files, which can look like valid
files, up to the point where a cluster was overwritten later. If you
miss to detect the file as deleted, you'll spend much time in trying to
find the no longer existing continuation clusters.

Post by Tonny Iversen
I don't know the specifics of jpg encoding, but maybe someone here
knows if the different parts of the image part of the jpg files are
numbered in some way, or contains some patterns/fields that can be
used to connect later fragments together with the first fragment? If
not, maybe something like FFT/Fouier transformations/frequency
analysis or something similar/else could be used to do a qualified
guess about which fragments belong to each other and maybe present
the user with some alternatives?

This requires a "map" of all clusters, containing the characteristic
information of each cluster. I made such a map already myself, for
restoring segmented text files by manually pairing the ends and begins
of the clusters. But with image data? At least you'll need an file
checker, that can determine whether a chain of clusters *might* be a
valid jpeg file.

Post by Tonny Iversen
I guess that if image files are fragmented on a memory card, the
chance is pretty high that the files only has two fragments. At least
this is my guess if the card was new or formatted before use, then
used to take pictures and erase some in between, and then
quick-formatted afterwards. And if the directory entries are intact
and most files are defragmented, I guess it could be possible also to
do an educated guess about which clusters are fragments belonging to
other files.

FAT32 allocates files strictly sequentially, until the medium is full.
You can assume that no overlapped writes occured, to multiple files at
the same time. In this case a recovery will work quite well. When files
have been erased, and the clusters have been reused later, you'll have
much more problems.

DoDi

[cmyk]

Post by Hans-Peter Diettrich
FAT32 allocates files strictly sequentially, until the medium is full.
You can assume that no overlapped writes occured, to multiple files at the same time. In this case a recovery will work quite
well. When files have been erased, and the clusters have been reused later, you'll have much more problems.
DoDi

Hi Hans,

Under the FAT system, if you delete a file, as one might do with a poor shot, then it's space is simply marked as being available
for re-use. If that file happens not to have been the last one written, then there is a good chance that either the next file
written to the media, or the one after that, will be fragmented. For example (in simple terms, ignoring cluster sizes), if the
deleted file occupied 1000K and the next two files written each occupied 800K, the second of those files would be fragmented into
200K and 600K segments. Similarly, if the next file written was 1200K, it would be split into 1000K and 200K segments. Given the
variability of jpeg file sizes and the tendency of many users to delete files when they review them, fragmentation over 3 or more
clusters is quite possible.

Cheers

[Hans-Peter Diettrich]

Post by cmyk
Under the FAT system, if you delete a file, as one might do with a poor
shot, then it's space is simply marked as being available for re-use. If
that file happens not to have been the last one written, then there is a
good chance that either the next file written to the media, or the one
after that, will be fragmented.

FAT32 appends files until the end of the medium is reached. This is an
optimization to prevent file segmentation. Deleted files are overwritten
only after the medium is full.

DoDi

[Phil Carmody]

Post by Hans-Peter Diettrich

Post by cmyk
Under the FAT system, if you delete a file, as one might do with a
poor shot, then it's space is simply marked as being available for
re-use. If that file happens not to have been the last one written,
then there is a good chance that either the next file written to the
media, or the one after that, will be fragmented.

FAT32 appends files until the end of the medium is reached. This is an
optimization to prevent file segmentation. Deleted files are
overwritten only after the medium is full.

I can think of few techniques _more_ likely to cause fragmentation!
(Assuming that's what you meant by "segmentation".) Every temporary
file (just think of your browser cache etc.) will create a hole, for
example. Every hole is a place where the end of one file and the
start of another, or even worse the middle of a file, will later be
placed.

Phil

[Hans-Peter Diettrich]

Post by Phil Carmody
I can think of few techniques _more_ likely to cause fragmentation!
(Assuming that's what you meant by "segmentation".)

Right, I meant fragmentation.

Post by Phil Carmody
Every temporary
file (just think of your browser cache etc.) will create a hole, for
example. Every hole is a place where the end of one file and the
start of another, or even worse the middle of a file, will later be
placed.

I suppose that image files are written sequentially, one after another.
If not, the problem is very hard (NP?).

DoDi

[cmyk]

FAT32 appends files until the end of the medium is reached. This is an optimization to prevent file segmentation. Deleted files
are overwritten only after the medium is full.
DoDi

FAT32 does not do this - that is a task for the OS or, perhaps, the media controller. Whether a particular camera or memory card
controller would do this when writing to the media is entirely up to the card or camera's maker. I know of none that have published
anything to the effect that their's behaves this way. In fact, for digital media, doing so is pointless from an optimisation
standpoint.

[Hans-Peter Diettrich]

Post by cmyk

Post by Hans-Peter Diettrich
FAT32 appends files until the end of the medium is reached. This is an
optimization to prevent file segmentation. Deleted files are
overwritten only after the medium is full.
DoDi

FAT32 does not do this - that is a task for the OS or, perhaps, the
media controller.

Read the available documentation about the FAT32 related data structures.

DoDi

[cmyk]

I did.

There is nothing in the FAT32 data structure that dictates the behaviour you claim.
--
cmyk

Post by Hans-Peter Diettrich

Post by cmyk

Post by Hans-Peter Diettrich
FAT32 appends files until the end of the medium is reached. This is an
optimization to prevent file segmentation. Deleted files are
overwritten only after the medium is full.
DoDi

FAT32 does not do this - that is a task for the OS or, perhaps, the
media controller.

Read the available documentation about the FAT32 related data structures.
DoDi

[Hans-Peter Diettrich]

Post by cmyk
I did.
There is nothing in the FAT32 data structure that dictates the behaviour you claim.

Look at FSInfo.FSI_Nxt_Free.

DoDi

[cmyk]

fsi.FSI_Nxt_Free specifies the lowest free cluster - there is nothing requiring this to occur after the last used cluster. It is
simply there to speed up disk writes on magnetic/optical media.

[Hans-Peter Diettrich]

Post by cmyk
fsi.FSI_Nxt_Free specifies the lowest free cluster - there is nothing
requiring this to occur after the last used cluster. It is simply there
to speed up disk writes on magnetic/optical media.

And consequently files are written contiguously, until the medium
becomes full.

DoDi

[cmyk]

Post by Hans-Peter Diettrich
And consequently files are written contiguously, until the medium
becomes full.
DoDi

Not necessarily if the lowest free cluster occurs before the last occupied cluster.

[Phil Carmody]

Post by Hans-Peter Diettrich

Post by cmyk
fsi.FSI_Nxt_Free specifies the lowest free cluster - there is
nothing requiring this to occur after the last used cluster. It is
simply there to speed up disk writes on magnetic/optical media.

And consequently files are written contiguously, until the medium
becomes full.

Are you sticking by your previous
"""
Deleted files are
overwritten only after the medium is full.
"""
claim?

If so, consider this sequence:
1) Create A
2) Create B
3) Delete A
4) Create C
After stage 3, what is the lowest free cluster? Is it after B,
or where A was?


If not, then can you point out where you retracted the claim?
And maybe why you held that point of view in the first place.
And what made you change your mind.

Phil

[Hans-Peter Diettrich]

Post by Phil Carmody
Are you sticking by your previous
"""
Deleted files are
overwritten only after the medium is full.
"""
claim?

Yes, of course :-]

Post by Phil Carmody
1) Create A
2) Create B
3) Delete A
4) Create C
After stage 3, what is the lowest free cluster? Is it after B,
or where A was?

Why bother with the lowest free cluster?

Of course it's where A has been, but that doesn't affect the write
pointer, which stays behind B.

DoDi

[Phil Carmody]

Post by Hans-Peter Diettrich

Are you sticking by your previous """
Deleted files are
overwritten only after the medium is full.
""" claim?

Yes, of course :-]

1) Create A
2) Create B
3) Delete A
4) Create C
After stage 3, what is the lowest free cluster? Is it after B, or
where A was?

Why bother with the lowest free cluster?
Of course it's where A has been, but that doesn't affect the write
pointer, which stays behind B.

Each of the above operations is atomic, so what the fuck is a write
pointer? You're barely making sense, so it's hard to argue against
you.

The FAT32 system apparently (see posts by other posters upthread)
writes each new file at the lowest cluster it can. That's from
where A has been. This is in agreement with all the prior FAT
systems that I have been familiar with.

If you can't work out how and where you're wrong given this
information, I simply don't have the time and patience to spoon
feed you any more. Perhaps try on comp.i.dont.have.a.clue ?

Phil

[Hans-Peter Diettrich]

Post by Phil Carmody
Each of the above operations is atomic, so what the fuck is a write
pointer? You're barely making sense, so it's hard to argue against
you.

You have read about FSInfo.FSI_Nxt_Free?

Isn't that a write pointer?

Post by Phil Carmody
The FAT32 system apparently (see posts by other posters upthread)
writes each new file at the lowest cluster it can.

So you think that the system scans the FAT, each time a cluster has to
be allocated to an file or directory?

Post by Phil Carmody
This is in agreement with all the prior FAT
systems that I have been familiar with.

Then familiarize with the new features of FAT32, before posting nonsense ;-]

DoDi

[cmyk]

Post by Hans-Peter Diettrich
Then familiarize with the new features of FAT32, before posting nonsense ;-]

And if you took the time to study how FSI_Nxt_Free is implemented, you wouldn't keep posting stuff like this. FSI_Nxt_Free points to
the lowest available cluster. Nothing more & nothing less. Nowhere is is said that there can't be any used clusters after this. Get
it?

[Hans-Peter Diettrich]

Post by cmyk

Post by Hans-Peter Diettrich
Then familiarize with the new features of FAT32, before posting nonsense ;-]

And if you took the time to study how FSI_Nxt_Free is implemented, you
wouldn't keep posting stuff like this. FSI_Nxt_Free points to the lowest
available cluster. Nothing more & nothing less. Nowhere is is said that
there can't be any used clusters after this. Get it?

"Typically this value is set to the last cluster number that the driver
allocated. "

Which part of that description did you not understand?

Hint: "allocated" is not "released".

DoDi

[cmyk]

"Typically this value is set to the last cluster number that the driver allocated. "

Which only goes to prove my point - 'typically' is not mandated behaviour and, as I said from the outset, that "is a task for the OS
or, perhaps, the media controller" - FAT32 does not do this of itself. Why don't you pay attention???

[Phil Carmody]

Post by cmyk

"Typically this value is set to the last cluster number that the driver allocated. "

Which only goes to prove my point - 'typically' is not mandated
behaviour and, as I said from the outset, that "is a task for the OS
or, perhaps, the media controller" - FAT32 does not do this of
itself. Why don't you pay attention???

Ah, so they're covering their backs. Their "typical" behaviour
is of no practical use in a scenario where the total amount of
data written is larger than the capacity of the storage medium,
and if anyone complains about that they can just say "well,
we didn't force you to use that technique".

OK, I admit it, MS have made their file system design worse
over time in this regard. However, they're at least not hiding
the fact.

However, as you say, they certainly aren't mandating the
behaviour claimed by HPD.

Didn't IBM teach them about proper file systems over a decade
ago?

Phil

[Phil Carmody]

Post by Hans-Peter Diettrich

Post by Phil Carmody
Each of the above operations is atomic, so what the fuck is a write
pointer? You're barely making sense, so it's hard to argue against
you.

You have read about FSInfo.FSI_Nxt_Free?
Isn't that a write pointer?

Given that one term has already been coined for it, there's
no need to coin another one. Economy of nomenclature.

Post by Hans-Peter Diettrich

Post by Phil Carmody
The FAT32 system apparently (see posts by other posters upthread)
writes each new file at the lowest cluster it can.

So you think that the system scans the FAT, each time a cluster has to
be allocated to an file or directory?

My god, you really are as stupid as I first suspected!

No I do not think that at all.

What on earth made you think a scan would be required every time?
That's what maintaining such an index is for, so you don't have
to scan. Think about which operations could make that index change,
and whether the system knows what it would change to after
completion of those operations.

Post by Hans-Peter Diettrich

Post by Phil Carmody
This is in agreement with all the prior FAT systems that I have been
familiar with.

Then familiarize with the new features of FAT32, before posting nonsense ;-]

Care to provide citations for where the standards deviate in
this regard?

Damn, I'd forgotten that I said I wasn't going to waste time
spoon feeding someone who has painted himself from head to toe
in KlooRezist. Due to lack of compression content, I'm removing
comp.compression from followups so I don't have to suffer your
no-doubt inane and ill-conceived reply.

Phil

[zac.badenoff]

Post by Tonny Iversen
Does anyone know of any software doing something like this
or is anyone considering making something like this?
I'm to pressed for time to consider developing something
like this right now, but if someone else wants to look into this,
I'd be more than happy to be involved in discussions around
the development if you want (though not very much since I don't
have much time available). And if someone could comment on these
issues, it would be great if I or someone else should consider
developing something like this.

Tonny,

'ere ya go matey!

It's been done so there's no need to try and develop something yourself.

http://www.ontrackdatarecovery.com/

Cheers,

__
zb

[Rob]

Post by zac.badenoff

Post by Tonny Iversen
Does anyone know of any software doing something like this
or is anyone considering making something like this?
I'm to pressed for time to consider developing something
like this right now, but if someone else wants to look into this,
I'd be more than happy to be involved in discussions around
the development if you want (though not very much since I don't
have much time available). And if someone could comment on these
issues, it would be great if I or someone else should consider
developing something like this.

Tonny,
'ere ya go matey!
It's been done so there's no need to try and develop something yourself.
http://www.ontrackdatarecovery.com/
Cheers,
__
zb

Someone looking for ontrack recovery

found a link http://www.lemonshare.net/Software/1782.html

[George Johnson]

Tonny Iversen wrote:

| Hello. I'm trying to figure out if it exists any software that might
| be able to recover fragmented images from flash memory cards /
| digital cameras if the card was quick-formatted (i.e. if the card has
| a zeroed out FAT), or some details about the jpeg algorithm to figure
| out how difficult development of such would be.
|
| My main interest is for .jpg/.jpeg files.
|
| In the situation when the FAT contains only zeroes, recovery of
| non-fragmented image files seems to be easy for many (or maybe
| most?/all?) file / image /data recovery software products, but I
| haven't been able to find any that can recover anything more than the
| first fragment of the file if the file is fragmented.

TESTDISK

It saved my hard drive when the boot blocks went screwy for no logical
reason a while back. I am replying off the very same hard drive that
TESTDISK recovered. Best of all, it's open source and thusly free to use.

Don't thank me, just be thankful that there is a kind soul out there
that put the effort into creating such a wonderful program that you can use
for free.

==============

To recover lost pictures or files from digital camera or harddisk, run the
PhotoRec command.
http://www.cgsecurity.org/wiki/PhotoRec

PhotoRec is file data recovery software designed to recover lost files
including video, documents and archives from Hard Disks and CDRom and lost
pictures (thus, its 'Photo Recovery' name) from digital camera memory.
PhotoRec ignores the filesystem and goes after the underlying data, so it
will still work even if your media's filesystem has been severely damaged or
re-formatted.

PhotoRec is free, this open source multi-platform application is distributed
under GNU Public License. PhotoRec is a companion program to TestDisk, an
app for recovering lost partitions on a wide variety of filesystems and
making non-bootable disks bootable again. You can download them from this
link.

For more safety, PhotoRec uses read-only access to handle the drive or
memory support you are about to recover lost data from. Important: As soon
as a pic or file is accidentally deleted, or you discover any missing, do
NOT save any more pics or files to that memory device or hard disk drive;
otherwise you may overwrite your lost data. This means that even using
PhotoRec, you must not choose to write the recovered files to the same
partition they were stored on.

Operating systems

PhotoRec runs under

* DOS/Win9x
* Windows NT 4/2000/XP/2003
* Linux
* FreeBSD, NetBSD, OpenBSD
* Sun Solaris
* Mac OS X

and can be compiled on almost every Unix system.
Filesystems

Photorec ignores the filesystem, this way it works even if the filesystem is
severely damaged.
It can recover lost files at least from

* FAT,
* NTFS,
* EXT2/EXT3 filesystem
* HFS+

ReiserFS includes some special optimizations centered around tails, a name
for files and end portions of files that are smaller than a filesystem
block. In order to increase performance, ReiserFS is able to store files
inside the b*tree leaf nodes themselves, rather than storing the data
somewhere else on the disk and pointing to it. Unfortunately, PhotoRec isn't
able to deal with this, it's why it doesn't work well with ReiserFS.
Media

PhotoRec works with HardDisks, Cdrom, Compact Flash, Memory Stick,
SecureDigital, SmartMedia, Microdrive, MMC, USB Memory Drives...
PhotoRec has been successfully tested with the following Digital Camera

* Canon EOS300D, 10D
* HP PhotoSmart 620, 850, 935
* Nikon CoolPix 775, 950, 5700
* Olympus C350N, C860L, Mju 400 Digital, Stylus 300
* Sony DSC-P9
* Praktica DCZ-3.4
* Casio Exilim EX-Z 750

Known file format

PhotoRec searches known file header and because there is no data
fragmentation (usually), it can recover the whole file. Photorec recognises
numerous file format including ZIP, Office, PDF, HTML, JPEG and various
graphics file formats. The whole list of file formats recovered by PhotoRec
contains more than 80 file extensions.

================

http://www.cgsecurity.org/wiki/TestDisk
TestDisk home: http://www.cgsecurity.org.


is OpenSource software and is licensed under the GNU Public License (GPL).

TestDisk is a powerful free data recovery software! It was primarily
designed to help recover lost partitions and/or make non-booting disks
bootable again when these symptoms are caused by faulty software, certain
types of viruses or human error (such as accidentally deleting your
Partition Table). Partition table recovery using TestDisk is really easy.

TestDisk can

* Fix partition table, recover deleted partition
* Recover FAT32 boot sector from its backup
* Rebuild FAT12/FAT16/FAT32 boot sector
* Fix FAT tables
* Rebuild NTFS boot sector
* Recover NTFS boot sector from its backup
* Fix MFT using MFT mirror
* Locate ext2/ext3 Backup SuperBlock

TestDisk has features for both novices and experts. For those who know
little or nothing about data recovery techniques, TestDisk can be used to
collect detailed information about a non-booting drive which can then be
sent to a tech for further analysis. Those more familiar with such
procedures should find TestDisk a handy tool in performing onsite recovery.
Operating systems

TestDisk can run under

* DOS (either real or in a Windows 9x DOS-box),
* Windows (NT4, 2000, XP, 2003),
* Linux,
* FreeBSD, NetBSD, OpenBSD,
* SunOS and
* MacOS

Source files and precompiled binary executables are available for DOS,
Win32, MacOSX and Linux from the download page
Filesystems

TestDisk can find lost partitions for all of these file systems:

* BeFS ( BeOS )
* BSD disklabel ( FreeBSD/OpenBSD/NetBSD )
* CramFS, Compressed File System
* DOS/Windows FAT12, FAT16 and FAT32
* HFS and HFS+, Hierarchical File System
* JFS, IBM's Journaled File System
* Linux Ext2 and Ext3
* Linux Raid
o RAID 1: mirroring
o RAID 4: striped array with parity device
o RAID 5: striped array with distributed parity information
o RAID 6: striped array with distributed dual redundancy
information
* Linux Swap (versions 1 and 2)
* LVM and LVM2, Linux Logical Volume Manager
* Mac partition map
* Novell Storage Services NSS
* NTFS ( Windows NT/2K/XP/2003/Vista )
* ReiserFS 3.5, 3.6 and 4
* Sun Solaris i386 disklabel
* Unix File System UFS and UFS2 (Sun/BSD/...)
* XFS, SGI's Journaled File System

[h***@gmail.com]

After reading your situations – I have found the following things: You want to recover deleted or lost JPG/JPEG image file format from flash memory card and for that you are looking for suitable Photo Recovery software.

Well, with this Best Photo Recovery Software, you can efficiently retrieve deleted, lost, erased, missing, disappeared photos (JPG/JPEG, PNG, BMP, PSD, etc) from flash memory cards.

Learn Detail Information About Photo Recovery Software From Here: http://www.rescuedigitalmedia.com