User:Skierpage/Disk recovery

Disaster!

I followed instructions to create a bootable SD card,

 zcat blah.bootable.gz > /dev/sdX

This is dangerous because it overwrites the partition, but I had done it before, I knew there was nothing on my SD card and /dev/sda is my SD card.

But /dev/sda is my hard drive. The whole thing.

I checked with df and the size looked wrong, then I read my notes (should have done that first) and I was overwriting the wrong disk. Ctrl-C!

Then I tested the .gz file and it was corrupt, so I thought nothing had been written. My computer worked fine.

The next day I got an error coming out of standby about not reading C:\$MFT or something. So I decided to reboot. Big mistake, I should have read partition tables, backed up some key files, and prepared for disaster.

Rule 1: do what you can while it's working.

Errors shutting down, then could not reboot, bad disk.

Failed recovery

I couldn't find my Windows XP CD-ROM, even though Falcon Northwest make a big deal. Of course my Windows XP is so dramatically different thanks to all the updates and service packs that the original disk has been useless for System File Checking and file recovery and such. I had always meant to follow instructions to slipstream a new boot CD-ROM, where you make a fresh Windows CD-ROM that contains the latest files, but never did.

Rule 2 You need a boot disk around.

Falcon Northwest provided me a personalized Recovery Disk, I inserted this and nothing happened. Zero feedback.

Into Kubuntu Live CD

I had built a Kubuntu Live CD. That booted fine.

The Kubuntu Live CD startup menu has "check disk" menu choice is useless, it just checks the CD-ROM. BUG: it is badly named.

So I ran the Live CD environment, and that worked fine. I'm updating this web page from the Live CD. Hooray for free software that is happy for the world to make bootable CDs.

Create a recovery disk/USB?

There are lots of dedicated "recovery CD" distributions. Distrowatch lists dozens, and the search leaves out the GParted distribution.

The problem is, I'm running from my only CD drive, so I can't burn another CD. Besides, I don't want to waste another CD blank. I have gigabytes of space on my USB flash drive and SD memory card.

BUG: All the distributions I checked have crap guides to creating a bootable USB. Some even talk about making a floppy. The functionality and documentation for making a bootable USB needs to be separated from all these distributions.

Use what I have

But I'm already running Linux. Kubuntu has the parted partition editor and the NTFS tools.

BUG: all the online web recovery instructions are crap. They tell you how to make a recovery CD or floppy or whatever, tell you to stick it in the machine and reboot, but fail to tell you what to do next. I guess they assume you'll walk through their beautiful menu system. It is stupid and short-sighted not to mirror help and guidelines on a web site. You should never have to create physical media and boot from it just to read documentation.

The first four pages of Google results for "linux partition recovery tutorial" are either sites trying to sell me something or yet another guide to creating a recovery CD. I gave up finding any explanation of what to do.

Find your hard drive

You have to find the UNIX device name for your disk, like hdb ("hard disk B") or sda ("SCSCI disk A" but also "Secure Digital" memory card). Then tools can access your disk using its entry in the special /dev directory. That's how I was able to scribble all over my hard drive in the first place.

Unfortunately, the /dev directory has lots of devices in it.

Another approach is to look for boot messages where the Linux kernel finds hardware devices. In a Linux console, enter

 dmesg | grep disk
 dmesg | grep drive

one of those should identify promising disk devices, or enter

 dmesg | less

to read the whole boot thing (which might have hints about the state of the drive).

The best approach I've found is the somewhat undocumented one of looking in the special file /proc/partitions:

 cat /proc/partitions

This identifies the partitions on the various devices, and their blocks. I guess if your hard drive was really badly trashed, it wouldn't have any partitions.

All these confirmed my hunch that /dev/sda is my poor hard drive. The good news is /proc/partitions showed my sda1 and sda2

Run parted

Now start parted. I should probably start this as the root user, using sudo

$ parted
Using /dev/sdb

that's wrong, I need to select /dev/sda.

 (parted) help
 ...  lots of commands
 (parted) select /dev/sda
 Error: Error opening /dev/sda: Permission denied

so I do have to be root.

$ parted
GNU Parted 1.8.8
Using /dev/sda   (good, now I'm root it picks sda as I can write to it)
Welcome to GNU Parted! Type 'help' to view a list of commands.
 
(parted) check
Partition number? 1
Error: \8. is 512k, but it has 15994 clusters (63976k).
Ignore/Cancel? i
Fatal: Bad FAT: cluster 12 is cross-linked for \0.8.  You should run dosfsck or scandisk.

there you go! Let's see if the Linux partition is OK at least.

(parted) check
Partition number? 1
 Error: Partition doesn't exist.

Damn.

 (parted) print
Model: ATA ST3120026AS (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
 
Number  Start   End     Size    Type     File system  Flags
 1      17.4kB  2031MB  2031MB  primary  fat32        boot, lba

damn. That's completely wrong, I should have an enormous Windows partition. My original destructive zcat bootable scribbled a new partition table on the front of my disk that's completely wrong.

How bad is it

So as a result of scribbling over the front of the disk I have

1. probably trashed the special loader (called grub? maybe) that let me boot from Windows or Linux
2. probably trashed the Windows boot code, I'm not sure if it's in the first partition or somewhere else (the MBR "Master Boot Record" ?)
3. probably trashed a big chunk of my NTFS Windows partition
4. somehow completely messed up my Linux setup. With firefishe's advice I had set up the second half of the disk with root, swap, and home, now it's nowhere to be seen :-(

parted has a rescue command that will try to locate and revive partitions, but I thought there's a backup somewhere on the disk with partition information. The "MBR" again?? But since the first partition table looks OK, the program trusts it.

Recovering partitions

So now I'm back to trying to find out more about disk repair.

• http://sysblogd.wordpress.com/2008/01/05/data-recovery-with-linux-from-hard-drives-cd-roms-or-dvds/ seems useful, has other links.

Fortunately I made notes when I created the partition setup.

/dev/sda1  ntfs  Size 90025 MB (83.8 GB) hopefully untouched
/dev/sda2 ext3 Mount / Format? checked  Size 20003 MB Used 495 MB
/dev/sda3 ext4 Mount /home Format? checked  Size 8003 MB Used unknown
/dev/sda4 swap Mount [blank] Format? [blank] Size 1998 MB Used unknown

Rather than guess at partitions with parted's rescue, testdisk sounded promising. But it's not on the Kubuntu live CD.

I tried using aptitude to install it, but that didn't work on the LiveCD. So I downloaded it. There's a Ubuntu package available, navigate to the Jaunty version, get the URL of the amd64-bit version. I have no idea how you can install new software to a read-only CD-ROM, so I did this in the temporary directory

cd /tmp
wget http://ubuntu.cs.utah.edu/ubuntu/pool/universe/t/testdisk/testdisk_6.10-1_amd64.deb

then

sudo dpkg -i testdisk_6.10-1_amd64.deb

to install.

testdisk to the rescue!

Praise be, this has a pretty good intro, so I walked through that while running

sudo testdisk

and then turned on its expert mode and went slowly.

Like every other tool testdisk initially found only the bad small partition:

Disk /dev/sda - 120 GB / 111 GiB - CHS 14593 255 63
Current partition structure:
     Partition                  Start        End    Size in sectors
 
Warning: Incorrect number of heads/cylinder 63 (FAT) != 255 (HD)
Warning: Incorrect number of sectors per track 62 (FAT) != 63 (HD)
 1 * FAT32 LBA                0   0 35   246 237 22    3966909 [LIVE]

But then its quick analyze command found my various Linux partitions!

And then, I pressed Enter then chose [Deep analyze], which went back through my disk slowly. Glory be, this found evidence of the bigger partition!

Disk /dev/sda - 120 GB / 111 GiB - CHS 14593 255 63
     Partition               Start        End    Size in sectors
D FAT32                    0   0 35   246 254 63    3968021 [LIVE]
P HPFS - NTFS              0   1  1 10944 254 63  175831362 [Windows XP]
D Linux                 1902   0  1  4419 254 63   40451670
 * Linux                10945   0  1 13376 254 63   39070080
D Linux                12139   2  1 14571 254 63   39086019</span
 P Linux                13377   0  1 14349 254 63   15631245
 P Linux Swap           14350   0  1 14592 254 63    3903795

wahoo! There's my Windows XP partition! It also found two extra bogus partitions, but by matching the size ("sector" is 1/2 kB) and using the info testdisk prints as you choose each partition, I figured out the proper partitions whose Start follows the End of the previous, and used the left and right arrow keys to enable the good partitions as shown, and delete the bad ones.

Press Enter, confirm writing the new thing. I'm not sure about the geometry errors.

Then it immediately displayed information about my NTFS partition:

Boot sector
Status: Bad
 
Backup boot sector
Status: OK
 
Sectors are not identical.
 
A valid NTFS Boot sector must be present in order to access
any data; even if the partition is not bootable.

so I chose its [Backup BS] to copy the backup boot sector.

I tried [Repair MFT] to check the NTFS Master File Table, but that requires rebooting.

One last check of the format (using the Advanced menu, confusing), and I exited and prepared to reboot!

Testdisk glitches

Quit for previous menu is odd

[Quit] everywhere to return to previous menu is very confusing. It should support [Esc]ape key, and the command should be [Back]

Analyze forces a Quick search

Also, if you choose analyze, you have a [Quick search] option, so there's no way to return to the main menu. You have to start a quick search. If you stop that, it'll be half-way through finding the partitions, so it looks as if it's lost all the information I've carefully brought back.

It's crazy that the only way I can see the current MBR is by choosing "Advanced".

So the main menu should have a "Current status" that shows

1. what's on the disk MBR and partition info
2. indicator if what testdisk has in memory is different (because you haven't written it yet)
3. if not, show what's in its idea of the MBR and partitions is
4. any other info it has or hasn't written to disk.

Ctrl-C

I tried to copy some text to here with Shift+Ctrl+C, and accidentally pressed Ctrl+C and testdisk immediately exited. Please trap Ctrl+C and ask "Are you sure?"

give user-friendly MB/KB size

When people are looking for partitions, they probably know the size in kB or kibibytes, but not in sectors. So maybe in parentheses give a user-friendly size like 5,203 KB or 12,304 MB.

Add partition is confusing

This is weird. First, I thought this would add the partition that testdisk had found that I've highlighted with the cursor keys, since D(eleted) is the opposite of Add.

Then, when I really did want to add a partition, the information line has no explanation

  D No partition             0   0  1  1020 251 62   15952104

and I can't choose a particular partition type with the cursor, it highlights a row of three at a time!!

List under [Analyze] > [Boot] is confusing

I re-entered TestDisk to view some files on my corrupted partition. I remembered this feature was called [List], but I had trouble finding it. I finally found it under [Analyze] > [Boot] , but the help for [Boot] is "Boot sector recovery". There's no indication that the [Boot] screen also has a "[List] directories and files, copy data from NTFS" feature.

Back to booting

I told TestDisk that Linux was my boot partition, however booting from the hard drive was still a complete failure, it just hangs forever. I have my partitions, but I've lost any knowledge of booting. So I had to boot the Kubuntu LiveCD again.

Kubuntu had installed the grub boot menu that gives me a choice of whether to boot Windows or Kubuntu, so all I need to do is create another one of those.

As usual there are tons of guides to restoring grub, and they're all crap out-of-date stuff that mostly explain nothing. E.g. my hard drive is /dev/sda, but grub turns this into hd0. And unlike TestDisk there's no way to know what the heck you just did or verify what you put on the disk. It's a shot in the dark.

https://help.ubuntu.com/community/RecoveringUbuntuAfterInstallingWindows is probably the best instructions, so I followed that.

Linux boot back

Now my hard drive boots into Kubuntu, but seems to ignore windows.

grub woes

The grub command is very poorly explained. I want a program to tell me what the current boot menu in my MBR looks like without rebooting. Does grub do this? Who knows. It seems to show a typical boot menu, but maybe that's coming from my Linux drive. Who knows?

Windows partition corrupt

Kubuntu automatically mounted my NTFS partition as it did before using the NTFS-3G driver:

 /dev/sda1 on /media/WindowsCdrive type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096)

but that was giving me tons of

ls: cannot access /media/WindowsCdrive/AUTOEXEC.BAT: No such file or directory
ls: cannot access /media/WindowsCdrive/boot.ini: No such file or directory

as well as a directory listing. Strange, since I was able to navigate in TestDisk to some of these files in "My Documents".

• fsck doesn't work yet for ntfs-3g mounts.

• There's an ntfsfix command, but it's in a different older NTFS for Linux package.

• Can I run Windows' chkdsk command under Wine?

• I found my Windows XP CD-ROM and booted into its Recovery Console, but that doesn't acknowledge I have a hard drive at all. It's supposed to ask what Windows installation I want to work with, but doesn't.

I asked about my problems on the NTFS-3G forum.

Back up the partition

I bought a Toshiba 400GB External USB 2.0 Hard Drive Model: HDDR400E03X , so I can just dump the entire NTFS partition onto it, insert it into a Windows laptop, and run chkdsk on it.

But how to "dump"? I think the command is

 dd if=/dev/sda1 of=/dev/sdc2 conv=noerror,sync

but I'm unclear how to create the NTFS partition.

Repartition USB hard drive

My hard drive according to TestDisk is

CHS 14593 255 63
1 * HPFS - NTFS              0   1  1 10944 254 63  175831362 [Windows XP]

TestDisk thinks the Toshiba has the same heads/sectors, so each "cylinder" is the same. fdisk -l thinks the Toshiba is

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       48641   390708801    c  W95 FAT32 (LBA)

• Clonezilla can clone partitions, but it's yet another boot CD to burn and a layer of software above.
• ntfsclone from NTFS tools can clone NTFS, but the doc doesn't make it clear whether it can clone to another partition.
• parted has a cp command, but the features page indicates that this doesn't work to create or copy NTFS partitions.

So I guess I'll create an NTFS partition on windows. I used parted to resize the FAT32 partition smaller to make room:

(parted) unit cyl
(parted) resize 1 0 20000
(parted) print
Model: Toshiba External USB HDD (scsi)
Disk /dev/sdc: 48641cyl

Number  Start  End       Size      Type     File system  Flags
 1      0cyl   19999cyl  19999cyl  primary  fat32        lba

(parted seems to count from 0, not from 1.)

Now to create an NTFS partition like

1      32256B         90025689599B   90025657344B  primary  ntfs         boot

on Windows, I called it partition X:. 90025657344 bytes is 87915681 kiB. I guess I'll round up to 86GiB for safety. That was a stupid idea because it makes it hard to copy back.

Backing up

From Kubuntu,

dd if=/dev/sda1 of=/dev/sdc2 options blah blah

worked fine.

Then reinserted disk in Windows. Explorer convinced the drive is broken; but from the command line I was able to run

chkdsk /f X:

lots of errors but it all looked pretty good.

Reinserted in Kubuntu, realized it was too big to copy back plus I want to make yet another backup for safety.

Permissions messed up

Reinserted in Windows to reduce file size. But first, a weird problem. Computer Management > Disk Management shows X: drive "86.00GB, Healthy (Active, Primary Partition)", 20GB free.

But Windows Explorer thinks X:\ is not accessible, access denied. Finally figured out that the ACL for it only allows administrator access!!

Compared with C security

Authenticated Users

Special permissions Allow grayed out

SYSTEM

Full Control, Modify, Read & Execute, List folder contents, Read, Write all Allow

Administrators (Machine Name\Administrators)

same as SYSTEM

Users

Read & execute, List folder contents, Read

C: Advanced security

Administrators (Machine Name\Administrators)

Full Control apply to this folder, subfolders, and fileto

SYSTEM

same as Administrators

Users (Machine Name\Users)

Read & execute apply to this folder, subfolders, and files

Authenticated Users

Special apply to Subfolders and files only

Authenticated Users

Create folders/append data, apply to This folder only.

Compared with new NTFS partition' security

Authenticated Users

Modify, Read & execute, List folder contents, Read, Write

SYSTEM

Full Control, Modify, Read & Execute, List folder contents, Read, Write

Administrators (Machine Name\Administrators)

same as SYSTEM

new empty NTFS partition's Advanced security:

Administrators (Machine Name\Administrators)

Full Control

SYSTEM

Full Control

Authenticated Users

Modify

Users (Machine Name\Users)

Read & execute

all applying to this folder and subfolders.

Started applying permissions to X: but got errors as it trundled through sub-folders setting permissions for SYSTEM:

cygwin\bin\get , \cygwin\etc\passwd , \cygwin\etc\profile.mytemp, \cygwin\etc\sshd_config, \cygwin\etc\ssh_host_dsa_key, more ssh, \cygwin\home;

\cygwin\lib\perl5\5.8\CPAN\Config.pm{,~}, \cygwin\lib\perl5\5.8\{Auto, Bundle, \cygwin\auto, \cygwin\file, \cygwin\HTML}, site_perl\Date blah blah more 5.8 perl stuff

\cygwin\usr\local\bin\mergebox.pl, \cygwin\usr\local\src

Reset it to make Advanced security match "empty NTFS partition" (i.e. not quite the same settings for Authenticated Users as on the laptop C: drive. Same errors.

This is probably why I can't access the drive from Recovery Console.

Make another backup partition...

Just to be safe I backed up again. I want to make an exact copy of /dev/sda1, but parted is hella confusing because

• the first partition doesn't actually start at the very start, it starts many bytes in to skip the boot record stuff
• parted indiscriminately rounds without telling you, e.g. "Each cylinder is 8225kB", actually each cylinder is 8225280 bytes.
• parted's math still doesn't work out. 10944 cylinders with geometry of 255 heads on 63 512 byte sectors is 90,017,464,320 bytes, but parted says my original partition is 90,025,657,344 bytes, quite a bit bigger. Arghhhh.

Number  Start     End       Size      Type     File system  Flags
1      0cyl      10944cyl  10944cyl  primary  ntfs         boot 
Number  Start          End            Size          Type     File system  Flags
1      32256B         90025689599B   90025657344B  primary  ntfs         boot 

I guess I'll work in bytes. Remove the existing end of Toshiba

(parted) rm 3

create new partition starting at the same place but with an end that's exactly the right size in bytes.

(parted) mkpart primary ntfs 256847642624B 346873299967B

check

Number  Start          End            Size           Type     File system  Flags
1      32256B         164505599999B  164505567744B  primary  fat32        lba
2      164505845760B  256847642623B  92341796864B   primary  ntfs         boot
3      256847642624B  346873299967B  90025657344B   primary  ntfs

partition 3 is exactly the right size.

Now make the remaining space another partition:

(parted) mkpartfs primary fat32 346873299968B 400085844479B

Number  Start          End            Size           Type     File system  Flags
1      32256B         164505599999B  164505567744B  primary  fat32        lba
2      164505845760B  256847642623B  92341796864B   primary  ntfs         boot
3      256847642624B  346873299967B  90025657344B   primary  ntfs
4      346873299968B  400085844479B  53212544512B   primary  fat32        lba

Now we can repeat the dd. parted has its own

 cp [FROM-DEVICE] FROM-NUMBER TO-NUMBER   copy file system to another partition

command, but stick to dd. From a guide, try bigger block size and force all blocks to be read and written:

sudo dd if=/dev/sda1 of=/dev/sdc3 bs=64M conv=sync,noerror