Hi Clauss, yes that seem to be the case, ‘cause the previous error seem to be not related to TWML.
I’ll have a look into how to make it more specific to TWML current script.
Thank you!
2 Likes
I haven’t got time to read the whole thing here, but I have installed on Flame 2021.2.2, centos 7.4, cuda driver 10.0.
Will it work on this version?
I installed and tried timewarping a clip, but the progress bar in a shell stayed at 0%
Hi Paul, it’s been developed on Centos 7.6 and currenlty the only two Flame boxes I have access to are both 2022.1 one on CentOS 7.6 and another on CentOS 8.2.
It looks like there’s a need to downgrade PyTorch library version to v1.2.0 - that was the one with Cuda 10.0. support.
To do it assuming you have an internet connection from the box:
init Miniconda3 environment using init.py script from the package and then
pip3 uninstall torch torchvision
pip install torch==1.2.0 torchvision==0.4.0
Feel free to PM me if you’re stuck,
Regards,
Andriy.
Hi Andriy, I can’t find the init.py script in the downloaded package.
Once the downloaded script got itself unpacked and installed it should be in ~/flameTimewarML/bundle folder,
It is actually calls init_env
Hi, just had a chance to check v0.4.3 of the script on the same config as yours:
v2022.1 on Linux, CentOS 8.2, HP G8
It did work fine with no issues so far. The recent discovery though was that I’ve made a mistake in exception catching and currently TimewarpML seem to be catching any error that happens in any python script and takes the blame on itself ) So, just out of interest, would it be possible to try and temporary run it without other scripts installed just to see if it is actually has something to do with it?
I’ll try to do it when I have a chance.
Would like to message another round of thanks here. I recently needed to deliver a 29.97 master on a show that was forced to shoot 23.98 on an XR stage. The MLTW completely saved the day.
6000 frames in total passed through. 97% success rate. Nothing works this good. The single shot it couldn’t handle was ironically fine in Kronos… for once.
Well done.
4 Likes
So here’s a quick question. I’m getting errors on one of my a6000 RTX boxes…. Fair to assume this is an Ampere related issue or is there something else going on @talosh?
Trained model loaded: /opt/Autodesk/flame_2022.1/flameTimewarpML/bundle/trained_models/default/v2.4.model
('CUDA error: no kernel image is available for execution on the device in ’
“<class ‘RuntimeError’>”)
[‘Traceback (most recent call last):\n’,
’ File ’
‘"/opt/Autodesk/flame_2022.1/flameTimewarpML/bundle/inference_sequence.py", ’
‘line 306, in \n’
’ I1 = F.pad(I1, padding)\n’,
’ File ’
‘"/opt/Autodesk/flame_2022.1/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", ’
‘line 3553, in _pad\n’
’ return _VF.constant_pad_nd(input, pad, value)\n’,
'RuntimeError: CUDA error: no kernel image is available for execution on the ’
‘device\n’]
Traceback (most recent call last):
File “/opt/Autodesk/flame_2022.1/flameTimewarpML/bundle/inference_sequence.py”, line 306, in
I1 = F.pad(I1, padding)
File “/opt/Autodesk/flame_2022.1/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 3553, in _pad
return _VF.constant_pad_nd(input, pad, value)
RuntimeError: CUDA error: no kernel image is available for execution on the device
Press Enter to continue…Exception ignored in thread started by: <function build_read_buffer at 0x7fcc62219af0>
Traceback (most recent call last):
File “/opt/Autodesk/flame_2022.1/flameTimewarpML/bundle/inference_sequence.py”, line 110, in build_read_buffer
frame_data = cv2.imread(os.path.join(user_args.input, frame), cv2.IMREAD_COLOR | cv2.IMREAD_ANYDEPTH)[:, :, ::-1].copy()
TypeError: ‘NoneType’ object is not subscriptable
You might need to update pytorch inside miniconda environment. There’s init_env script to get it initialized in bundle folder, and then try “pip install torch --upgrade”
1 Like
The issue seems more complex. First konsole wasn’t installed. Upon installing konsole and reinstalling I get:
(‘Traceback (most recent call last):\n’
’ File “/opt/Autodesk/shared/python/flameTimewarpML.py”, line 421, in ’
‘unpack_bundle\n’
’ payload_file.write(base64.b64decode(payload))\n’
’ File “/opt/Autodesk/python/2022.2/lib/python3.7/base64.py”, line 87, in ’
‘b64decode\n’
Getting the same. Just installed an A5000 and got this from my all time favourite tool:
Maybe it helps?
Received 1 clip to process, press Ctrl+C to cancel
initializing Timewarp ML…
Trained model loaded: /home/Ton/flameTimewarpML/bundle/trained_models/default/v2.4.model
(‘CUDA error: no kernel image is available for execution on the device in ’
“<class ‘RuntimeError’>”)
[‘Traceback (most recent call last):\n’,
’ File “/home/Ton/flameTimewarpML/bundle/inference_sequence.py”, line 306, ’
‘in \n’
’ I1 = F.pad(I1, padding)\n’,
’ File ’
‘"/home/Ton/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", ’
‘line 3553, in _pad\n’
’ return _VF.constant_pad_nd(input, pad, value)\n’,
'RuntimeError: CUDA error: no kernel image is available for execution on the ’
‘device\n’]
Traceback (most recent call last):
File “/home/Ton/flameTimewarpML/bundle/inference_sequence.py”, line 306, in
I1 = F.pad(I1, padding)
File “/home/Ton/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 3553, in _pad
return _VF.constant_pad_nd(input, pad, value)
RuntimeError: CUDA error: no kernel image is available for execution on the device
Press Enter to continue…Exception ignored in thread started by: <function build_read_buffer at 0x7f993eb05d30>
Traceback (most recent call last):
File “/home/Ton/flameTimewarpML/bundle/inference_sequence.py”, line 110, in build_read_buffer
frame_data = cv2.imread(os.path.join(user_args.input, frame), cv2.IMREAD_COLOR | cv2.IMREAD_ANYDEPTH)[:, :, ::-1].copy()
TypeError: ‘NoneType’ object is not subscriptable
I have no such card in vicinity but I believe the only way it could be fixed it to update the PyTorch library to its recent version.
In order to do so one need to init miniconda python environment first, and then run "pip3 install torch —upgrade” (assuming the machine is connected to internet, if not there is a way to download package and show pip where it is located).
There’s init_env script in the bundle that would init the environment, but it does use konsole app that is not installed by default on Centos8, it is very easy to change to gnome-terminal or anything though.
Here is a command that should initialise the environment:
/bin/bash --rcfile <(echo ‘. ~/.bashrc; eval “$(~/flameTimewarpML/miniconda3/bin/conda shell.bash hook)"; conda activate’)
You should see (base) before your command prompt that would indicate that.
Then try to upgrade PyTorch with pip3 install torch —upgrade
I just did it on CentOS8 system I have acesss to and it did upgrade torch to version 1.10.0 (the bundled one was 1.7.1)
Let me know if it works
2 Likes
Do you mind to download the v0.4.3 release from here: Release flameTimewarpML v0.4.3 · talosh/flameTimewarpML · GitHub
replace the python file in /opt/Autodesk/shared/python (or wherever it is installed), move ~/flameTimewarpML folder out of the way (just rename it to .old or smth) and start flame to perform clean install again.
after it is installed one can init base conda environment with:
/bin/bash --rcfile <(echo ‘. ~/.bashrc; eval “$(~/flameTimewarpML/miniconda3/bin/conda shell.bash hook)"; conda activate’)
you sould see (base) before your command prompt.
the try to run “pip3 install torch —upgrade”
and then start flame again and check if it does solve it
Shell returns: Illegal variable name.
Does it mean after updating PyTorch to 1.10 the TimewarpML still does not work?
Ok @talosh I removed the Konsole -e line from the init_env script, here’s what I did…
./init_env
(base) bash-4.4$ pip3 install torch --upgrade
Collecting torch
Using cached torch-1.10.0-cp38-cp38-manylinux1_x86_64.whl (881.9 MB)
Requirement already satisfied, skipping upgrade: typing-extensions in /opt/Autodesk/flame_2022.1/flameTimewarpML/miniconda3/lib/python3.8/site-packages (from torch) (3.7.4.3)
Installing collected packages: torch
Attempting uninstall: torch
Found existing installation: torch 1.7.1
Uninstalling torch-1.7.1:
Successfully uninstalled torch-1.7.1
Successfully installed torch-1.10.0
(base) bash-4.4$ exit
exit
Upon running this is what the shell throws out…
'CUDA kernel errors might be asynchronously reported at some other API ’
‘call,so the stacktrace below might be incorrect.\n’
‘For debugging consider passing CUDA_LAUNCH_BLOCKING=1.\n’]
Traceback (most recent call last):
File “/opt/Autodesk/flame_2022.1/flameTimewarpML/bundle/inference_sequence.py”, line 306, in
I1 = F.pad(I1, padding)
File “/opt/Autodesk/flame_2022.1/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py”, line 4174, in _pad
return _VF.constant_pad_nd(input, pad, value)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Press Enter to continue…Exception ignored in thread started by: <function build_read_buffer at 0x7f989fdd3ca0>
Traceback (most recent call last):
File “/opt/Autodesk/flame_2022.1/flameTimewarpML/bundle/inference_sequence.py”, line 110, in build_read_buffer
frame_data = cv2.imread(os.path.join(user_args.input, frame), cv2.IMREAD_COLOR | cv2.IMREAD_ANYDEPTH)[:, :, ::-1].copy()
TypeError: ‘NoneType’ object is not subscriptable
If you run nvidia-smi what cuda version is reported?
There might be a need to specify cuda version for pytorch on upgrade
@talosh Driver Version: 460.91.03 CUDA Version: 11.2
Ok. Success. The two changes were editing away “konsole -e” ini the init_env script and then running
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
This is on an a6000rtx and 2022_2. Thanks @talosh for all your help.
2 Likes