Digikam/Tutorials/Switch from Microsoft OneDrive or WLPG to digiKam

From KDE UserBase Wiki
Revision as of 18:00, 31 December 2021 by Yurchor (talk | contribs) (Fix minor typos)

Introduction

This tutorial is for people who would like to use open-source digiKam in-parallel or instead-of some Microsoft software that organizes photos (especially if you use people face tags):

DigiKam is an excellent replacement for discontinued WLPG and can even be made cross-compatible to use both digiKam and Microsoft software interchangeably (see configuration tutorial) but you should proceed with care and caution when you introduce digiKam or any other software that can alter your existing metadata.

In addition to this tutorial it is highly recommended to also be familiar with related tutorials by the same author: “Setup of digiKam for Windows compatibility” and “Tagging and Face Tags in digiKam”.


Back it up! Seriously

Before you allow digiKam (or any other metadata editor) import (show) your previously tagged image files, make sure you have taken all the precautions:

  1. You have to back-up all your existing image & video files, preferably so that you can very quickly copy them back to the main location (you may need to do it multiple times).
  2. Create a test photo gallery folder where you will copy-in a few files to test in digiKam, before you give digiKam access to your real photo gallery.
  3. Make an additional Permanent backup of photo metadata – see below. This is to save you from any issues that may not even be related to migration to digiKam.
  4. Test as much as possible before you accept that migration was successful and consider overwriting your older backups with digiKam modified files:
    1. Try tagging over 100 photos will all types of tags that you use.
    2. Try searching for photos using different tags (hierarchical, star rating, etc.).
    3. Check how files from different cameras, different sources and formats have migrated.
    4. Make sure you know what will happen with video metadata, people tags on rotated pictures and other potential issues (see "Microsoft image metadata" section).
    5. If you plan to use cross-compatibility with OneDrive tagging, WLPG or other software – try to repeatedly make edits in one program then observe the changes in another so that you have files where each type of metadata was over-written in for example:
      • WLPG → digiKam → WLPG → digiKam
      • WLPG → digiKam → WLPG → digiKam
      • (there should not be any lost, duplicated or abnormal tags created from any of the edits. Exception - digiKam will likely create descriptive tags for all People Face Tags. This is not a significant issue for cross-compatibility.)
  5. Keep your Microsoft Photo Gallery installed as long as possible. This is one more advantage of cross-compatible digiKam setup – you should be able to use old Photo Gallery in parallel to digiKam in case you need to access some video metadata or any other reason.

Permanent backup of photo metadata

Here is an easy method to make a complete metadata backup from all your photos that will easily fit on one M-Disk DVD or a cheap flash drive. You never know when some virus, child or silly mistake will irreversibly wipe or corrupt your metadata. If you notice the damage too late, all your backups may by then already hold the corrupt files and only such permanent backup will be able to help. It is not only good for tags you add manually, but also to preserve more basic metadata like the date when a picture was taken.

The basic principle is that metadata, including all camera data, tags and people face tags are completely resolution-independent – you can dramatically resize and compress image data while preserving 100% of the metadata. Such conversion must be doable by many ways, including in digiKam (please edit this tutorial to add your favorite method), but my personal favorite tool for such metadata backup is:

Method A
  1. Open the free "FastStone Image Viewer" for Windows so that you see the main toolbar (not full-screen mode). This program is reliable and has minimal risk of messing up your files.
  2. Go to Tools → Batch Convert ... and "Add" all your photo gallery folders to the "Input List".
  3. Next to “Output Format” set JPEG and in the “Settings” choose quality for example 70%. (we want images to be recognizable, not beautiful).
  4. Under "Use Advanced Options" choose “Resize based on long side” to for example 1024px. (again just to keep images recognizable, you can go lower if you need to save more space).
  5. Under “Rename” it is nice to add for example 
    *_metadata_bckp
    to each file, so that you better recognize the highly compressed files if you will work with them.
  6. Choose a reasonable output folder and press Convert (folder structure will be preserved).
  7. When everything is done, I suggest you add a text file note with date when you did the metadata backup and for what reasons. Then compress everything with 7zip to reduce the size even more and have one easy to move file for permanent storage.

Note

Such batch conversion in "FastStone Image Viewer" will not back up metadata of videos and likely will skip RAW files. We assume that the key metadata is stored in normal files such as .jpg or in the file and folder names. If you have sidecar .xmp files, they would need a different backup approach, but for backing up from Windows Live Photo Gallery, this method is almost perfect.


Microsoft image metadata

Before you start using digiKam to manage your WLPG tagged gallery, it is good to understand where the potential risks and issues are. Depending on how you used tags in WLPG, some of these concerns may not be applicable to you!

  1. Tag hierarchy needs care – Default Metadata settings in digiKam are prone to create many loose tags out of the nicely organized hierarchies such as for example “Location/USA/New York/Central Park”. If you set up digiKam correctly ("Setup of digiKam for Windows compatibility"), this will be 100% resolved.
  2. Face / People tags are a challenge – digiKam can fully supporting people tags as they were handled in WLPG, but again for backwards compatibility to Microsoft software, you must have correct settings in digiKam. More importantly – because Microsoft is the only company in the world that does not correctly support image rotation tags, any face tags done in WLPG on rotated photos will appear in digiKam wrongly rotated. It really is a Microsoft bug that we need to live with and the issue is covered in detail further below.
  3. No Flag – setting “flag” in WLPG is not saved to any file (only internal database), so this data is lost any time you move to another computer or another program [source].
  4. No metadata for videos. In contrast to photos, common video file formats do not support embedded metadata. As a result WLPG only stores video metadata in the internal database without any method to export the data Therefore WLPG video metadata is lost any time to you move to another computer or another program. DigiKam actually solves this issue by creating extra .xmp sidecar files next to each un-writable file, so as long as the extra files are not lost, digiKam will always preserve your video tags in the future. In fact, to be safe, it is a good idea to always keep year-month-day date and some keywords in the filename of important video files, because it is easy to separate video file from its metadata.
  5. Beware of "Read-Only" files! – It is worth double-checking that you do not have any Read-Only files and folders in your collection. We must have all metadata stored inside the files for digiKam to read it, but no software will write your metadata into files if the files are write-protected.
    • In case, you just changed some read-only files to writable, you need to open WLPG and to be safe - add some new tag to all files that were previously Read-Only. Wait for WLPG to write that tag along with all other metadata to all files (then you can easily remove that dummy tag in WLPG or later in digiKam). + you can always double-check which metadata is saved in a file by opening file properties in Windows Explorer.
  6. The rest of the metadata is straightforward. There is little risk that captions, star ratings and individual descriptive tags from WLPG or OneDrive would get lost. In fact, most software can deal with these more basic tags and you may find other software, such as built-in Windows tools as good-enough alternatives to digiKam which still save metadata into files.

For more information you are very welcome to read excellent articles by José Oliver-Didier which cover a lot more about Microsoft image metadata specifics: https://jmoliver.wordpress.com/2017/02/12/accessing-windows-photo-gallery-metadata-using-exiftool/ , https://jmoliver.wordpress.com/2020/02/16/onedrive-photo-captions-and-tags/ and other related blog posts.

Tip - use digiKam to examine your metadata structure.

digiKam offers excellent built-in ExifTool integration in the right side "Metadata" panel that can be configured in the digiKam settings to show 100% of the Metadata in your images. Together with "XMP" and "IPTC" tabs in the same "Metadata" panel you have a convenient tool to see how the imported metadata looks when digiKam opens your old tagged file for the first time and then see which metadata is written by digiKam after you try to change old tags and add new ones. (ideally, we want digiKam to overwrite all existing metadata fields when you make an edit - so that file does not contain both current and left-over obsolete Tags.)


Microsoft Face tag rotation issue

Microsoft’s face tags (usually from WLPG) are affected by long-term-bug which results in face regions being shown rotated by 90 degrees to one side on some of the pictures. Ideally, there should be a method to permanently fix the face tag rotation issue for all image files and make the results cross-compatible, but it may also be enough to just fix each affected file one by one.

Root cause and potential workaround

As long as the image “Exif orientation tag” is not “Normal”, Microsoft WLPG will always treat face tags differently than all other software that recognizes “Exif orientation tag” (for example digiKam). Therefore to make a file universally interpreted by all software including Microsoft WLPG any affected file needs to be losslessly rotated so that “Exif orientation tag” is re-set to “Normal”.

In practice, the problem is almost exclusively present in portrait orientation images with face tags (but only if the image contained “Exif orientation tag” which is typically written by digital cameras with an orientation sensor - again not in all cameras).

Diagnosing the extent of the problem

Using ExifTool it is possible to generate a list of photos that will have rotated face-tags due to broken Windows Photo Gallery image orientation/rotation code. This list is great to understand the extent of the problem in your gallery and to find example files to test any fixes.

  1. Get "exiftool.exe". All operations are executed in Windows usual Command Prompt (cmd.exe) when you have "changed directory" to the folder where the exiftool.exe is located (similar command will work also in Linux and MacOS).
  2. Generate a list of photos that will have rotated face-tags. Note this command just scans photos - NO CHANGES to your FILES will be made after this command. 
    exiftool -T -Filepath -Orientation -RegionRectangle -r "D:\FolderNameWhereYourPhotosAre" > PhotoOrientationReport.txt
  3. You can import this .txt file into Excel or LibreOffice as a Tab-separated list. And apply a filter so that:
    1. In column “B” deselect “Horizontal” and “Unknown”.
    2. In column “C” deselect the empty rows.
    3. Done – the remaining rows are your files that have face tag rotation issues.


The whole database solution

Currently, there is no simple solution to run and forget, but the solution is close... Most likely the solution can be implemented using ExifTool, but we may need some more brave testers to finalize the concepts. You can review these examples for initial guidance:

https://exiftool.org/forum/index.php?topic=4361.msg33803#msg33803

https://exiftool.org/forum/index.php?topic=12345.msg68463#msg68463

https://exiftool.org/forum/index.php?topic=12963.0 (making command time-bound)

Notice that even if you fix all your old files at some point, you may still get new affected images in the future, so it is not strictly a one-time issue for everyone. And remember to “Back it up! Seriously” whenever testing new scripts or fixes.

Fixing one file at a time

In case you have only a few affected files or the whole database fix is not practical to use, there are different methods to make corrections on an individual file – please see the tutorial on how to Rotate face tags on individual files.

Other Metadata issues

For most users, the instructions above will give great results and you do not need to read any further. In case you encounter specific problems, here are some extra hints:

Summary

Metadata is tricky – different software looks for different data and interprets it in different ways. Nevertheless, everyone can retain all their metadata from WLPG and other Microsoft software in digiKam. But with some additional digiKam configuration you will also retain cross-compatibility to continue searching through your metadata directly in Windows, OneDrive or any other Microsoft software.

Finally, if you make some configuration mistake – it is not the end of the world. Perhaps just using your previous backup will solve your issue, but even in really bad cases there are command-line tools such as ExifTool can come to the rescue.



About this tutorial

This tutorial was originally written for digiKam 7.4 in 2021-12. You are very welcome to edit, update and Upgrade this tutorial page to make it better and keep it up to date. No attribution or copyrights on this page (all illustrations are original as well)!

Retrieved from "https://userbase.kde.org/index.php?title=Digikam/Tutorials/Switch_from_Microsoft_OneDrive_or_WLPG_to_digiKam&oldid=470403"