As discussed on today’s Patron’s Chat, and as I foolishly promised to post, here is a fun trick for texture extraction and reprojection using a 3d track and action cameras. It’s a bacon saver.
Get yourself a good 3d track of your scene. Locators at points where you’d like to place surfaces are nice, but you can always use Find-A-Point to get them after the fact.
With your background selected in the action media list, attach a diffuse map to the surface, and set its mapping mode to Projection: Camera1 (or whatever your hero tracked 3d camera is called)
Attach a camera to the axis of your surface. Frequently the camera is on the bottom looking up, you may have to flip your surface around to have the camera looking down at the surface. You’ve just created a downshooting camera that will capture a static version of whatever it’s pointed at, which is very handy. Much more handy than what I used to do, which was track perspective grids to geometry like a fucking animal.
Create a second action output, make sure your surface and diffuse map are active (I typically just hit All Objects out of sheer laziness, but you can actually make as many cameras and outputs as you want from a single action, which is also quite handy).
Notice that on output 2, you now have a very stable (or as stable as your 3d track, I suppose) image/plate/texture, onto which you can place, say, a picture of @andy_dill or do whatever paint work you desire.
Duplicate your first action node (I like Duplicate With Connections for this, because it saves you a bunch of tapping and reconnecting. Connect the output of whatever work you’ve done on the flat image to the input layer formerly occupied by the color source.
Some things to keep in mind: The orientation of the original surface is pretty key, so fiddle with that while you scrub around on the camera2 output to make sure your texture is staying perfectly stable. Only after you’re satisfied with that orientation should you duplicate your action setup to reproject, because the source and destination surfaces need to be identical. You can scale and rotate the surface to make it work for your purposes, but you should make sure it’s pinned to the origin axis for the background, xyz should all be 0.
Finally, and most importantly, I got this from @p2FX so thank you again for that, Patrick. I don’t know where he got it, probably Dill, who no doubt discovered it on some manga forum somewhere. It’s my understanding that this is similar to something @DannyYoon has placed on the portal, but I haven’t tried that one and frankly have been too dumb to comprehend most of what Danny has told me for about 20 years now. I stand on the shoulders of giants, in case that’s not clear.
If I got any of that wrong or skipped any steps, I hope they will drag me mercilessly for it.
Thanks Kirk, that is awesome. This by the way is the feature request from the Feedback Forum. Don’t know if you can still upvote it, as it got accepted, but whatever helps…
FYI most of the time when someone posts a procedure they name the topic after the person that developed the procedure, not the whiny asshat that requested it.
@p2FX also showed me this trick a while back, which had me tickled… Attaching a camera directly to the surface… Who’da thunk? I always thought it’d be a lot more complicated voodoo to get the camera frustrum aligned to the surface.
It also works well with the new Camera Analysis but again, sticky plane is only as good as the track. YMMV.
Used it to extract a ground plane texture at edge of frame from a really distorted wide angle lens. Regular track wasn’t solving, but the Camera Analysis gave me something I could at least work with.
One thing I do cos I rarely get a perfect track is do a secondary 2d stabilize using planar tracking, uvs, 1pt, whatever, and then un-stabilize it before plugging it into the second action.
One thing I’d like to play around with is using this within an undistort/redistort workflow via syntheyes or similar, but my grasp on that process is still iffy at best.
Right! Using @bryanb ’s excellent UV stabilizing gag from yesterday is a great intermediate step for fixing all kinds of evil shit before reprojecting.
So I’ve come back to the page a few times and have never successfully made it work. @kirk I don’t suppose you have the setup for this shot so I can figure out where I’m going wrong? Or perhaps a video of a little step by step? I realise this thread is pretty old now haha so sorry for bringing it back up again.
Hello! No need to apologize! As luck would have it this is my favorite topic, plus it’s raining, so I’d be happy to whip up a setup and/or a video. But out of curiosity, where are you getting stuck?
I think i was falling short when i needed to attach the camera to the surface. I believe when i tried it i had been supplied a camera and geo - perhaps it’s because the axis for the geo didn’t have x / y values - can’t remember. Either way, i’d love to get this down cos stabilizing shit is my go to
If it it’s of any help, I just decoded these instructions for myself and got it to work with the Flame 3D tracker. Also tried to simplify a few things.
Here are the steps I used:
Create 3D Tracked scene
Create action, Camera Analysis (see Flame Learning channel if necessary)
With default camera selected, drag in Camera anslysis
Set reference frame, use ‘Analysis’
Make sure to set ground plane and origin
Select point cloud section in the area of interest
Create alignment axis in 3D space
use F8 to see camera view, ctrl-drag to select multiple points, surface will be positioned on averages of all selected points
use ‘Scene Tracker’ button to create axis, XY or XZ as need, adjust in object space if needed
Add 2nd input to action with white color source
Drag 2nd input into schematic to create surface and parent to scene tracker axis from previous step
Use the axis that came with the surface to scale and position it appropriate (this generates the alpha channel for the composite)
With background selected in media panel, right click on surface and add diffuse map
On diffuse map change mapping to ‘projection’ and then select the 3D tracking camera for the projection (right underneath)
With the surface selected, add a new camera node
In output, change the output camera to the newly added camera
As Kirk, mentioned the new camera may be looking from the wrong side, if so fix by changing axis that came with surface, to rotate 180deg for X y Z
Duplicate the action with connections
Change background to the first action
Change the input media to the replacement media
In output, change the output camera to the original 3D tracking camera
In output, enable the matte output to add alpha channel
Delete the diffuse map and it’s axis, as well as the 2nd camera
And turn ‘Back Off’ in media panel to only map projected media to output
Add comp node to composite second action over original footage
If instead of new footage you use texture to paint, place paint node between actions, or render out, process externally (Si, etc.) and then bring back in.
PS: In one instance the recording camera didn’t point at the surface correctly. I was able to address that by adding a ‘look at’ link as such (from the toolbox). Adding the camera it doesn’t parent to the axis any longer, but at the surface (this is all in 2023.3)
Note for the planar positioning happening around minute 6:
If you’ve got tracker points that are on the surface you are looking to place (as you do here) you can hang a sphere geo off all the axes that are on that plane. You may need to scale down the sphere geo, but you should get 3d spheres at each point on your planar surface.
Hang the projector card off of one of the planar axes with zero position transforms (as you did in the video), then manually rorate the card using the r-key and onscreen UI until the projector card is intersecting all the spheres. I suggest making spheres and card different colors so you can easily see when they are being bisected. This’ll get it into a reasonably coplanar position.
If you want to do futher non-rotation tweaking, add a downstream axis and only use the X&Y position, Z rotation, and X and Y scale.
@kirk great breakdown! Do you ever mess around with the “add card” function in Syntheyes to make geo planes? I had some excellent success with this when I was tracking like 10 tv screen comps a couple weeks ago. Game changing in terms of fast and easy planar alignment of geo for me.
