It's the sudden change in colour spectrum that is stressing the fish out.
In the morning when the sun comes up there is lots of yellow light and very little blue light shining down directly on us. There is blue light but due to the angle of the morning sun it's not that noticeable.
As the sun rises the amount of blue and red light increases and becomes more direct. By lunch time the blue, purple and UV light is directly above us and becomes quite intense.
As the day progresses, the angle of the sunlight changes and as the sun is going down, the amount of blue, purple and UV light decreases and more yellow light becomes visible.
If you have the blue, red and white lights on during the day, that gives the fish a natural colour spectrum for midday light. In the morning and evening you can reduce the white light and have red light only or red and blue. If you can't dim the white light separately, then just turn the lights off together.
Have the room light on before you turn the tank light off so the fish don't go from bright light to complete darkness. Then wait 30 minutes or more before turning the room light off.