Disappearing frames in Flame centralised framestore

Sysadmins
1

Hi everyone,

We recently implemented a centralised Flame Project Server and framestore in the studio and we’re running into some issues.

The setup:

  • Flame Project Server 2025.2.2 on VM
  • Framestore on TrueNAS ZFS NAS mounted with NFS
  • Flame 2025.2.2 on three P620 workstations

Reading and writing performance has been decent, even compared to our old local NVMe SSD RAIDs. We’ve structured it with one project per show, with workspaces per user.

Minor problems:

  • Batch clips in restored 2024.2.1 desktops sometimes show as checkerboard or lost after working fine initially.
    • Not a huge deal, we ended up doing restores from batch setups going forward.
  • Soft-imported clips that get restored from a batch setup show as checkerboard initially.
    • Not a huge deal, we can reimport these.
  • New soft-imported clips in batch setups will sometimes be perfectly fine initially, then later show as checkerboard. Terminal error when viewing: WD: :Cannot get rendered state of frames: Frame id not found. Cannot access Frame 12: Frame id not found
    • Sometimes this is fixed without intervention. Sometimes this is fixed by moving the clip to a shared library and back.

Major issue:

  • After a period of working without encountering these issues, yesterday we shut down the Flames and Project Server to restart the framestore. Today we find the following:
    • Some batch renders have checkerboarded
    • When attempting to view, we see in terminal something like:
      I/O on unreferenced frame /studio/flame/prod/8/0x4d0002af00000087.mio
Cannot access frame 20: General software error
WD: :Cannot get rendered state of frames: Frame id not found.
      or
      I/O on unreferenced frame /studio/flame/prod/5/0x4d0001cd0000001c.raw
I/O on unreferenced frame /studio/flame/prod/5/0x4d0001cd0000001d.raw
I/O on unreferenced frame /studio/flame/prod/5/0x4d0001cd0000001e.raw
I/O on unreferenced frame /studio/flame/prod/5/0x4d0001cd0000001f.raw
    • Navigating to these paths manually shows that no such file exists on the framestore, despite the render being perfectly usable yesterday.
    • Running a vic -v stonefs -s lost on project server to show lost frames, we’re seeing a large number of lost frames (anywhere from 400 to 28,000) reported across multiple clibs/projects, e.g.
      Checking libraries for lost frames ...
/opt/Autodesk/clip/stonefs/project.prj/artist.wksp/PREVIS.000.clib references 28111 missing frames.
      as well as this recurring error flooded throughout the shell while it runs:
      Could not set reference count for frame ids list: Frame id not found
Could not force reference counts: Frame id not found
Could not set reference count for frame ids list: Frame id not found
Could not force reference counts: Frame id not found
Could not set reference count for frame ids list: Frame id not found
Could not force reference counts: Frame id not found

Just wanted to pick your brains regarding these issues and whether anyone has encountered them, regardless of Flame version and local/remote/central framestore.

We thought maybe an automatic vic incorrectly detected/marked some frames as unreferenced and deleted them, but we’re unsure how this would be possible. Any other ideas?

Cheers

2

@danielcai

This is a support call - get that started now.

Flame Version, Project Version, Project Server Version, Networking, FileSystem Type, File Server Type are all incredibly important ingredients in this mix.

Who set this up and are they your support ticket liaison?

What is your identity management solution?

When the frames get written, can they be overwritten by another user on another flame?

1 Like
3

Out of curiosity is this a StorNext SAN?

4

@kirk

1 Like
5

Ha! Was just about the ask the same question, after THE @panisset shared this in the StudioSysAdmins discuss-workstations channel…

From JFP:
This is fairly specific, but if you are running recent-ish Flame versions with a framestore on a Stornext filesystem, you may have run across the issue where DPX files don’t read correctly and showed up as the dreaded checkerboard in Flame: the DKU driver package used to include an Autodesk “trick” version of the NVIDIA driver which had a one line fix in the DMA setup code, but at some point the DKU stopped including this fixed NVIDIA driver and you had to request it explicitly from Autodesk support.
The fix is apparently now in the vanilla 550.90.07 driver included with DKU 19.2, so there should no longer be issues with DPX files on StorNext filesystems.

image
image1300×193 39.9 KB

3 Likes
6

Yup. It certainly SOUNDS like the same problem. Though as @cnoellert points out, he specifically said he was running TrueNAS. Apologies for missing that part.

Hopefully support can get you sorted out. These kinds of problems are infuriating.

7

It also appears that these are raw files, so i suspect a floating point project and intermediates.
I’m not much of a gambler since i only have $0.02 but i suspect that it’s users writing over the top of other users.

8

In fact, the fix is on the StorNext 7.2. This is why we stopped providing modified NVIDIA driver since out friends at Quantum fixed the issue. The issue was :SNXT-1215 Autodesk Flame fails to render DPX file with error Bad Address

More info here:

What's New in StorNext 7.2*

@panisset could you add this information in the thread on the StudioSysAdmins discuss-workstations channel? Merci :wink:

1 Like
9

Apologies for the posting misinformation, I updated my post on StudioSysadmins.

Unfortunately StorNext updates are never simple, I know several facilities not on StorNext 7.2.x yet, so they will need to continue using the “custom” NVIDIA driver until they can update their StorNext metadata controllers (the rule with StorNext is that you should never have a client running a newer version of StorNext than the MDCs).

2 Likes
10

Merci!