Spent half of the week debugging strange GUI hangs, trying different desktop environments, graphics drivers, kernel versions etc.
In the end the cause for these hangs was a bug in the SSD Firmware. After upgrading the Firmware of the A400 SSD the issues went away completely. (Unfortunately this is only possible within Windows, and only when using the basic windows AHCI drivers - there is not even a DOS based updater)
tl;dr: when debugging performance issues consider hardware bugs as well.