For many years I’ve been using a Kindle Touch as my primary device to read books and long-form web content. I also employ a strict regimen to capture my learnings via my BookTracker app. Recently, I upgraded to a Kindle Paperwhite for its better screen resolution and warm backlight. I thought that the upgrade would be as seamless as that of a phone’s, and I could easily migrate all my notes and reading progress over.
I couldn’t be more wrong! Turns out it can be as easy as a snap, or as big an ordeal as moving your house. It all depends on where you source your content and how you got it to your Kindle in the first place.
Let me share with you the different migration scenarios and how you can save yourself from headaches along the way.
Different ways to migrate Kindle content
Your migration complexity is determined by where you get your content, and how you initially load your content into Kindle. There are 3 possible scenarios:
- buy ebooks directly from Amazon (either via website or directly from Kindle’s interface), which automatically sends the book to Kindle
- source your own content, then use the Amazon Send to Kindle feature to send it to Kindle
- source your own content, then “sideload” it from PC/Mac using the Calibre app or other offline tools via USB
Migrating content already in Amazon cloud
In scenarios 1 and 2, if you have previously turned on Whispersync in your old device, your annotations, highlights and bookmarks will be automatically uploaded to Amazon’s servers whenever the Kindle is connected to wifi. They will then be ready to sync to all other devices registered in the same account. That means if you register a new device (Kindle e-reader or Kindle phone/tablet app) with the same Amazon account, you can easily download the content with all the associated metadata to keep up with your reading progress. This is the simplest approach.
To make it clear, these are the metadata types associated with each book
- bookmarks are those you add via the upper right corner of the screen
- highlights are those you add by dragging and highlighting a block of text in the book
- annotation is your own written notes over a highlight
- time and location information for last page read, how much time you’ve spent and how much time to completion
As you can see from the screenshot below, you can login to your Amazon account, go to the Manage Your Content and Devices page, and select the books you want to deliver to your device. When this happens, both the book and its metadata will be transferred over. You can also do a Select All to transfer everything in a batch. While you’re there, you can also do some household organization by creating new collections and categorizing books into different collections. The changes you perform on the website will automatically be sync’ed to your devices when they go online.
You can also view and edit your notes and highlights here online.
Problems with migrating sideloaded content
The 3rd scenario is the tricky one I’m facing, since Amazon doesn’t track your sideloaded content, making this whole migration process a pure manual one.
After I received my new Paperwhite, I copied the whole /documents/ folder (containing all books) and the myclippings.txt file in the root directory of the old Touch to the Paperwhite. I thought that would be everything. I soon found out that all my sideloaded content, while being ported over successfully (i.e., they can be opened without a hitch), lost all their respective bookmarks and highlights. When I open a book in the Paperwhite, it appears as if the book has never been read. It starts from the beginning and there is no highlight or bookmarks!
Some of you might be familiar with the file myclippings.txt which you can easily open and view your highlights. I thought that would take care of everything. However, upon closer examination of the /documents/ folder structure, I found out that each book (let’s call it BookName) has its own /BookName.sdr/ folder, with a pair of files storing the book’s metadata such as last page read, % remaining and total read time, as well as your actual annotations and highlights.
You might think, since the /BookName.sdr/ folders are all copied over, shouldn’t the new device be able to read metadata from them and show each book’s highlights and time information correctly?
Well, not exactly. My next discovery is that, the pair of metadata files are not just named like BookName.azw3f and BookName.azw3r for an azw3-type book. Rather, they have the filenames of BookNameab12......89xy.azw3f and BookNameab12......89xy.azw3r. The ab12......89xy suffix turns out to be a unique ID for your Amazon account that you registered the Kindle with.
For example, the book The War of Art.azw3 will yield The War of Artab12345689xy.azw3r and The War of Artab12345689xy.azw3f in The War of Art.sdr folder. If your book is of the mobi format, then it will produce The War of Artab12345689xy.mbp1 and The War of Artab12345689xy.mbs. PDF and other formats will get their own pairs too.
How does this affect us? This means that if your new device is not registered to the same account as the old one, then you are in trouble. In my case, since my Paperwhite is not yet registered (and thus not associated with any Amazon account), when my Paperwhite opens The War of Art.azw3, it fails to see The War of Artab12345689xy.azw3r and The War of Artab12345689xy.azw3f as its corresponding metadata files. Instead, my Paperwhite will now create a new pair of metadata files called The War of Art.azw3r and The War of Art.azw3f, which as you can guess, are just a blank slate to begin with. Essentially, my old bookmarks, highlights and time info are all in the new device, but remain untethered from the book.
So why didn’t I register this new device to the same account? My problem is that my Touch was registered with Amazon China to buy Chinese ebooks. However, with the upcoming closure of Amazon China’s Kindle store, I don’t want to register my new device to the Amazon China account anymore.
How I finally recover my bookmarks and highlights
What finally solved my problem is to batch remove the ab12......89xy suffix that Kindle adds to all the existing metadata files. In this way, my Paperwhite treats the metadata as if it has created them natively itself. This only works if you have not opened any book you copied over. Otherwise, a new pair of files have already been created for you, and attempting to rename the old pair to strip the ID from filenames will creat a name conflict situation with the new pairs.
This solution will also work if you have devices registered to different accounts. Perhaps you registered your Kindle to a German Amazon account while working there, then when you moved back to the US, you got a new Kindle and registered it to a US account. Registered devices will automatically tag a unique ID (associated with your Amazon account) to all the metadata files the first time you open the book, while unregistered devices won’t. To share your content between devices that are registered to different accounts, just rename the ID in all files to the new ID you use. To find the new ID, open a book in the new device, then check the new metadata files in its .sdr folder to get the right suffix.
Cleaning up unused metadata files after migration
After I’ve successfully reconnected all existing bookmarks and highlights, I decided to take a deeper look at what’s in the /document folder. I then found out many .sdr folders that don’t have their associated books. These are metadata files left over when I deleted a book from the device.
The following table lists possible scenarios. o means the deletion is clean, that both the book and .sdr and metadata files are gone. x means only the book is deleted and both .sdr and metadata files are orphaned. Sideloaded books (and their metadata) are not saved to Amazon servers, so obviously you cannot delete them in Amazon site (the N/A situation).
| Methods | Bought from Amazon | Sent to Kindle | Sideload from Calibre |
|---|---|---|---|
| Delete in Kindle | x | x | x |
| Delete in Amazon site | x | x | N/A |
| Delete in Calibre | o | o | x |
The files, even though not big, will take up some space if you have a large collection and you frequently shuffle books in and out of your device. If you don’t anticipate reading the book again (loading it and retrieving the metadata files), then their presence makes no sense. Upon further research in various forums, users have recommended this tool to detect these leftover files and delete them.
Looking for a better workflow in knowledge management
This ordeal also got me into thinking that all these highlights and notes are useless without a structured way to view and edit them alongside my notes from paper books. Currently, the notes are in standalone silos, disconnected from each other and unable to form a cohesive knowledge system.
Previously, I’ve used Clippings to save and edit my highlights. One drawback is that it only reads from the MyClippings.txt file, and doesn’t touch annotations and bookmarks, which usually mark specific paragraphs or chapters that I want to revisit.
I’m now looking at Kmate for better indexing of highlights and annotations, as well as this or this tool to programatically extract highlights, bookmarks and annotations to feed to my notetaking pipeline. My uber solution is to automate all these so that all my highlights/notes are automatically exported to OneNote for editing, and then integrated with my BookTracker so that I don’t need to maintain separate systems for online and offline books. I’d also like to tag, index and connect my notes across books, so that regardless of where the notes originate from, they will be building blocks in different subjects I’m learning. Another weekend project in the horizon!