There’s so many issues that could be contributing to this. One thing that I find helps with edge issues is to use a luminance curves-based comp node for the problem area. In Flame, it’s called the Matte Curves node and in Nuke it’s called Addmix. There’s some directions on using the Flame one that I’ll link below but, frankly, the way I use it to just connect the inputs and just play with the curves until I have something that looks ok.
Aside from that, some other ways to deal with key issues: use your key as a matte to grade up/down the background in the area where you’re having an issue. Make sure the problem you’re having is actually in your comp node and not from some other distortion/lighting effect that’s part of your setup.
Possibly, using IBK might help.
Also, if you’re doing something like keying and comping everything with denoised plates (as you should be), the method you’re using to put grain back might be causing the issue.
I would suggest keeping your key and front image separate. Just take the matte output from your key and use that as the matte input in your comp node. And then take your image and pipe that into the front. You’re going to need to do some grading, despill and edge extension on the image, it looks like.
I don’t know if NeatVideo is causing this problem. I doubt it but it is interesting that you noticed the blue channel has more noise. This is common and one of the excuses I use for shooting against green rather than blue (sometimes).
It is caused by the uneven distribution of Red, Green and Blue photosensors on a camera image sensor.
This will probably not help you with this black edge but it is mildly interesting. When you solve the edge problem look up Bayer Filter
The advantage of testing this out using a log colourspace (Arri Wide Gamut Log C) as @randy mentioned is that it should (ACEScc and ACEScct still have negative values) remove all negative values.
not sure what color space you’re using but it looks like negative values what happens when you put a clamp node on the front and back see what that looks like , this will prevent negative values, but its just a test and don’t keep it that way but it will let you know if that’s what’s causing the problem, from the look of the front back it appears to be two separate color spaces and you have a transform happening creating negative values.