Longhorn metadata restoration
If you found this article, you probably lost your Longhorn metadata in one or the other way. This sounds scarier than it really is, because Longhorns metadata is actually really easy to re-create manually.
Warning: While this has worked for all my volumes in the past and I have not encountered any lasting effects or data-loss, I do not guarantee your success with this method. Please make appropriate backups of the files you are editing before trying this. This is a last-resort when you have no working replicas available and cannot attach the volume at all.
Diagnosing failed metadata
The most common symptom of failed metadata is a attach-detach loop of the volume. You can see this in the Longhorn UI by filtering for the status "In Progress"
Restoring volume metadata
Once you have found a problematic volume, click on the volume to open its more detailed page. Hover over the storage path of the replica and note yourself the full storage path shown in the tool-tip as well as the volume size. See the selected text in the picture for guidance on where to hover the mouse cursor. The replica below is not affected by metadata corruption and is only used for visual reference.
For the next step you will need a remote console session to the server on which the replica is stored on. Navigate to the full storage path you noted down in the previous step. If you have one or more of the following files you are in the right place:
- volume-head-###.img
- volume-head-###.img.meta
- volume-snap-*.img
- volume-snap-*.img.meta
- volume.meta
Check the filesize of the .meta files. If one of them is 0, this is the one you need to repair. Below are guides for each kind of metadata file, as they are different!
Repairing volume.meta
You lost the main metadata file for this volume, luckily this is fixable. First, you will need an example of a valid metadata file. I will provide you with one of mine:
{"Size":8589934592,"Head":"volume-head-001.img","Dirty":true,"Rebuilding":false,"Error":"","Parent":"volume-snap-51131b80-b84f-4a33-abaf-7f8ea1a4468e.img","SectorSize":512,"BackingFilePath":""}
First you will need to calculate the correct size of your volume. This can be done by multiplying SizeInGiB x 1073741824. My volume above is 8GiB. Replace the size in the metadata with the calculated size.
Next, find your volume-head-###.img and set it as the value for the Head field. For Parent, order your volume-snap-*.img by their timestamp and choose the latest one. Once you are done the file should look somewhat like this (those below are still fake-values!):
{"Size":YourVolumeSize,"Head":"volume-head-###.img","Dirty":true,"Rebuilding":false,"Error":"","Parent":"volume-snap-*.img","SectorSize":512,"BackingFilePath":""}
Once you saved the file, Longhorn should be able to attach the volume and begin to create new replicas for it.
Repairing volume-head-###.img.meta
This one is very similar to the volume.meta file, minus some of the status fields. Here is what a valid file would look like:
{"Name":"volume-head-001.img","Parent":"volume-snap-51131b80-b84f-4a33-abaf-7f8ea1a4468e.img","Removed":false,"UserCreated":false,"Created":"2021-07-25T23:44:51Z","Labels":null}
Find your volume-head-###.img and set it as the value for the Name field. For Parent, order your volume-snap-*.img by their timestamp and choose the latest one. Once you are done the file should look somewhat like this (those below are still fake-values!):
{"Name":"volume-head-###.img","Parent":"volume-snap-*.img","Removed":false,"UserCreated":false,"Created":"2021-07-25T23:44:51Z","Labels":null}
Once you saved the file, Longhorn should be able to attach the volume and begin to create new replicas for it.
Repairing volume-snap*.img.meta
This one is almost equal to the volume-head-###.img.meta file. Here is what a valid file would look like:
{"Name":"volume-snap-51131b80-b84f-4a33-abaf-7f8ea1a4468e.img","Parent":"","Removed":true,"UserCreated":false,"Created":"2021-07-25T23:44:51Z","Labels":null}
Remove the .meta from the filename and set it as the value for the Name field. For Parent, order your volume-snap-*.img by their timestamp and choose the one older than the snap you are currently fixing. If there is no older one, leave it empty. Once you are done the file should look somewhat like this (those below are still fake-values!):
{"Name":"volume-snap-*.img","Parent":"volume-snap-*.img OR empty","Removed":true,"UserCreated":false,"Created":"2021-07-25T23:44:51Z","Labels":null}
Once you saved the file, Longhorn should be able to attach the volume and begin to create new replicas for it.