What you drew:
There has been a recent thread about a Victron being disconnected from its batteries.
The instructions ask you to connect the battery first, and *then* connect the panels.
It's going to partly hinge on when/why you're flipping the master switch.
If it's only for rare maintenance, you're definitely OK (especially if you flip a blanket over a panel or two).
If you're flipping more frequently, you're
probably OK, as long as you don't flip it back on when the isolator is joining the systems with the alternator running OR if the panels are over 18 volts. If they have an open circuit voltage (Voc) of over 20 volts, the Victron may make the mistaken assumption that your (disconnected) batteries are really 24 volts.
In theory, you can manually *set* the Victron to 12 v (versus "automatic"), but it's something to watch for.
I forget the name of the recent (within a month) thread that was chasing this down, but it was a bizarre system with a 2nd solar controller (a Yeti Goal Zero) between the Victron and the battery.
Adding a switch or removable fuse between the panels and the Victron would really make it OK.
--dick