a/c wont stay on until the car warms up

i need help, i’ve been restoring all the luxuries these cars came with to use as a daily work beater, with the a/c being one of them. I installed everything, had the system vaccumed and filled with r134a. It works very well and is pretty cold but i have an issue. It wont stay on until the car warms up! i would hit the button and i hear it switch on and the idle drops but right away it goes back to normal. But once the car is warmed it then it kicks on and stays on and it starts to cool the air.

Any ideas?

any a/c experts here?