Hey @ALan ive updated Multi-gpu brench.
Changed things thinking on PCIe GPUs as well, how the weights are update and how data are sync between gpus. You should get better perfomance.
Now multi-gpu and single-gpu are merged into one, so you dont need anything else.
Tested on multi 6000 Ada cards and worked great, almost same speed but double batch size.
make sure to git clone from multi-gpu brench, main brench im keeping original for now.
Since i mainly use SXM cards and those automatic deal with p2p in between, i endup not paying attention to pcie, but now that is gone.
