I wanted to replace the 64 GB SD card in my Galaxy Note 4 with a 128 GB SD card. I assumed that I could simply take the smaller SD card out of my phone, connect it by USB (with a USB-based SD card reader) to my MacBook, copy the contents to the larger SD card, insert the larger SD card into my phone, and then go on my merry way.

Of course not.

When I inserted the smaller SD card into my MacBook, it told me that it could not read the SD card. I assume it is a formatting issue, so I guessed that I needed to transfer files by USB from the phone. I connected the phone by USB and launched Android File Transfer, but sadly, Android File Transfer is terrible. When I tried to copy a large amount of data at once — a single folder tree with 15 GB of audio in it — Android File Transfer froze up. I couldn’t tell the difference between “slow” and “stuck”. It seemed to handle smaller batches of data just fine, but I refuse to babysit these tools. The devices work for me.

Searching the web, I found a suggestion to use FTP to send the files from the phone to my MacBook. This meant installing AndFTP on the phone and launching ftpd on the MacBook. I tried that. Sadly, FTP is not a great protocol for “make these folders the same”. At best, it can “overwrite/resume”. When my phone didn’t copy all 58 GB of data in one session, I couldn’t tell how much it had copied and which files I needed to recopy. Moreover, it sent files at approximately 4 MB/s, which means that it takes about 4-5 hours to try to copy everything. This appears to be just long enough for copying to fail.

Srsly.

As a last-ditch effort, I tried installing adb so that I could copy files with a command-line tool, which is more likely to work and, more importantly, easier to script if I need to copy files in batches. Naturally, the Android world has decided to “make things easier” by repackaging the Android development tools inside Android Studio, thereby making it harder to learn how to use them as simple command-line tools.

It turns out that, at the time, I ignorantly went “the long way”, so I ended up doing all this:

  1. Launch Android Studio, since I had installed it previously in a desperate attempt to hack on some Android project about a year ago. (How do you think it went?)
  2. Notice that Android Studio is out of date, so update it, then restart it.
  3. Notice that Android Development tools are out of date, so update them, then restart Android Studio.
  4. Try to launch adb from a command line, and notice that the Android Development tools are not on the system path.
  5. Figure out where Android Studio installed the development tools, so that I can try putting that directory on the system path.

Finally, this seemed to help. I could launch adb from the command line.

$ export PATH=$HOME/Library/Android/sdk/platform-tools:$PATH
$ which adb
/Users/jbrains/Library/Android/sdk/platform-tools/adb
$ adb
<adb usage guide>

Yay! Now… how do I actually copy stuff?!

Copying Files from Android Phone With adb

I assume that I need to “start a debugging session” with adb by “mounting” my phone as a device, then I can use adb pull to pull the relevant files from my phone.

First, however, I need to enabling USB debugging mode on my phone. Fortunately, I have done this before. Doing this seems to depend on the device and Android version, so I recommend that you search the web to learn how to enable USB debugging mode on your device. For example, I searched for enable usb debugging mode samsung "galaxy note 4" and found http://www.solvemix.com/index.php/android-smartphones/315-samsung-galaxy-note-4-how-to-activate-usb-debugging-solved.

So far, so good. To recap:

  1. Enable USB debugging mode on the phone.
  2. Unlock the phone.
  3. Connect the phone by USB.
  4. On the MacBook, dismiss and close Android File Transfer, because it is useless.
  5. On the phone, allow the MTP connection.
  6. On the MacBook, dismiss and close Android File Transfer again, because it can’t take a hint.
  7. Check that adb can find the phone.
$ adb devices
List of devices attached
<meaningless hex number>    device

I assume that device is my phone, because what else could it be?

Now, can I pull a single file? First, let me poke around the phone.

$ adb shell
shell@trlecan:/ $ cd sdcard
shell@trltecan:/sdcard $ ls
<a listing of the files on my SD card>

Yay! I want to pull the directory tree /sdcard from my phone to my MacBook. May I?!

First, let me pull a single file.

$ adb pull /sdcard/temporary_contacts.txt ~/Desktop
[100%] /sdcard/temporary_contacts.txt
$ ls ~/Desktop | grep con
temporary_contacts.txt

Success! Now, clean up before moving on, then pull a directory of reasonable size.

$ rm ~/Desktop/temporary_contacts.txt
$ ls ~/Desktop | grep con
<nothing>
$ mkdir -p ~/Desktop/pull-directory && adb pull /sdcard/scanner ~/Desktop/pull-directory
<an actually useful progress report showing something that suggests that the computer is copying files to my Desktop!>

I stopped this job after it was 20% finished, because I felt convinced adb pull would do what I want it to do. It copied about 170 MB in what seemed like less than 1 minute. At 170 MB/minute, a full copy of 58 GB would take about 6 hours. I hope it’s faster than that.

Since I don’t know how long it will take to pull the entire 58 GB on the card, I start with a big-but-not-huge directory and time the pull operation, so that I know how long to expect the whole backup to take.

# Clean up before moving on
$ rm -fr ~/Desktop/pull-directory
$ mkdir -p ~/Desktop/pull-directory && time adb pull /sdcard/scanner ~/Desktop/pull-directory

Even though I’m using the time tool, I start a stopwatch on the side, just in case time somehow fails. Now I wait.

Woohoo! time worked and alleges that copying that directory took 4 minutes, 35 seconds, copying 1018 MB of data. My arithmetic says 3.70 MB/s, so 58 GB would take 16052 seconds or 4.45 hours to copy. I can live with it, as long as it actually works.

And… of course, the directory called /sdcard on my phone is in fact not the root of the SD card storage on my phone, because that would make sense and be helpful and the universe is not that merciful. How did I discover this? I tried to check the size of /sdcard and it reported something entirely unexpected.

$ adb shell
shell@trltecan:/data $ du -h /sdcard/ | tail -n 1
4.9G    /sdcard/

Uh… what?! I expect something closer to 58 GB, so why is there only 4.9 GB of stuff here?! Sigh. What is this directory and why is it on my phone? Is it an alias? (Later I found out that it is indeed an alias. Fine.)

Fine. Where is the SD card on my phone, O great Android?!

shell@trltecan:/data $ df
<a bunch of Permission denied messages>
Filesystem                       Size     Used     Free   Blksize
<a bunch of irrelevant file systems>
/storage/7C2F-B7C8              59.7G    52.3G     7.5G    128.0K
<a bunch of irrelevant file systems>

Aha! 52.3 GB used out of 59.7 GB. That must be it. I’m so glad that I wasted 5 minutes copying files from the wrong place onto my phone! At least now I can test again, in case my measurement of 3.70 MB/s changes due to copying from the SD card instead of my phone’s built-in storage. Let’s try that again.

# Clean up before moving on
$ rm -fr ~/Desktop/pull-directory
$ mkdir -p ~/Desktop/pull-directory && time adb pull /storage/7C2F-B7C8/scanner ~/Desktop/pull-directory

The directory /storage/7C2F-B7C8/scanner has 840 MB of data, which required 3 minutes, 25 seconds, for an average of 4.10 MB/s. Encouraging! At that rate, 52.3 GB would require about 13062 seconds or approximately 3.5-4 hours to copy. That would be nice!

Finally, I think, I can just do it.

# Clean up before moving on
$ rm -fr ~/Desktop/pull-directory ~/Desktop/BARNEY-SD
$ mkdir -p ~/Desktop/BARNEY-SD && time adb pull /storage/7C2F-B7C8/ ~/Desktop/BARNEY-SD

Now, we wait…

After We Wait…

It took 3 hours, 47 minutes and it seemed to work! The transfer ran at only about 4 MB/s, but at least I could more easily see progress and there’s no GUI to get in the way. Now I need to wait an hour or so to copy these files to my larger SD card so that I can install it in the phone. Now that I know how all this works, I can do this more easily next time.

Summary

Use adb to pull files from the SD card onto a new storage device, which could be the new SD card mounted on the computer.

  1. Enable USB debugging mode on the Android device.
  2. Install the Android Development tools, and specifically adb.
  3. Find the mount path on the device for the SD card using df.
  4. adb pull <mount path on Android SD card> <destination drive>, then wait several hours.

It sounds so simple, once you’ve actually learned how to do it.

But Wait! There’s More!

There’s no globbing in adb pull! If you want to copy a subset of the files in a directory, then you need to resort to lower-level trickery.

$ adb shell ls $1 | tr -s "\r\n" "\0" | xargs -0 -n1 adb pull

Here, $1 is your pattern for matching files, such as /storage/emulated/0/DCIM/Camera/20161207* to match all photos taken with your camera on December 7, 2016.

Thanks to http://stackoverflow.com/questions/11074671/adb-pull-multiple-files#11250068 and https://github.com/sschuberth/dev-scripts/blob/master/android/pull_all.sh for their help!