Having finally managed to get this panel off I was able to get rid of the rotten fabric.
My plan for this is to clean it back to the metal then paint it with a hammered black finish. I reckon that will look like it could have left the factory that way.
If it looks odd I'll look at getting the panel re-covered with the correct material.