Can i install open wrt in my Tp link Archer c86v1(US)?

Sorry to say it, no chance

4MB firmware file says it is not (a modern) linux.
Same proprietary "leopard_e" platform as here

i.e - no SoC support in linux, and the device is too small for any linux.

No need to bother with serial connection or finding boot log for same reason.