You can set the V-Ram allocation in the setup utility. It defaults to automatic, but you can set it to 80% or whatever you prefer. Was instructed to do that by support when we were recently chasing a GPU memory leak.
The other useful thing - open a terminal window and type
watch -n 0.5 nvidia-smi
It will update every 1/2s with actual GPU memory usage from the GPU.
A lot more detail about all of this here.
In my experience Flame honors those limits, until something goes haywire and then it just blows right past them.