How to avoid getting this flicker/discolouration when upscaling dark 2D animation? (check video link)

Link to the example video: https://www.youtube.com/shorts/FiPJvXXYSos

The balconies are black without using topaz however it looks like they light up and flicker after using sharpen/upscale. I'd appreciate if someone tells me how to get around this. I have tried lowering the amount of sharpen as much as I can and tried other models too but it seems to not be doing anything still getting this issue.