baloo stores tags in xattrs (extended attributes) on the filesystem.
Some filesystems (e.g. FAT32, which you would commonly find USB volumes formatted with) do not support xattrs at all, so copying those files to such a volume will trash the tags.
Most CLI utilities (e.g. cp, tar, rsync) won't copy xattrs without additional flags either, so unless your environment is 'All KDE all the time' and you only use Linux filesystems like ext4, XFS or BtrFS (this is not a complete list), and you modify your backups to specifically include xattrs, your tags will probably get trashed one way or another.
Nepomuk (previous KDE tagging/file metadata system) used to store tags in a separate database, but even after many years of development it could hardly be made to work performantly or reliably so it was replaced with baloo.
In some ways using xattrs is 'the right place' to put this tagging info, but Linux doesnt have a 'core API' that would allow a 'xattrs preserved by default, and interpreted consistently' policy. Therefore there are a lot of ways to lose your tags, and it may even be futile to begin with. KDE has been trying to make this work for many years, but it is difficult given the way UNIX/POSIX has evolved.