I think your config switch_port section lacks an option device switch0 to link it to the switch. Without that option, swconfig likely ignores this per-port section entirely.
This is not what I meant. I mean that everyconfig switch_vlan and config switch_port section requires an option 'device' 'switch0' to relate this section to the switch it belongs to.
You need
config switch_port
option device 'switch0' # this is mandatory
option port '4'
option pvid '2'