I would think you would have more potential leaks in the future when you disturb the seal at doors, the roof and the lower valance. I also think you would be hard pressed to get every screw hole to line up just right again.
I have taken some panels off mine from the inside and I don't think it is all that complicated, just might need to have some new batten strips on hand to finish the job. My interior vinyl wrap stuff is coming off due to excess heat in the summer breaking down the glue so I have been peeling it off and repainting.