Collect Media Script (aka archiving un-managed media)

Flame Python
1

For quite some time now people have asked for the an ability to collect clips similar to how AE does. I had a bit of spare time on my hands so I thought I’d give it a go. I must say, I have a new appreciation for the dev team who have to envision every possible scenario a user could organize things! Turns out nothing is all that straight forward.

What It Does

The script will go through all the non-shared libraries within the current workspace and scrape everything to find clips. This includes folders, reels, reel groups, batches and desktops. If a clip is uncached then it is added to a file (a note of this later). At the end you can use this file to run rsync to copy everything to a backup location of your choice. The original folder structure is kept.

As of version 0.5, there is the option to scrape only un-cached media or everything regardless if it’s cached or not. I figured this could be useful to some. You could use it to track everything used in a project for exmaple.

What Doesn’t Work/Cavets

  • Currently you can not get the cached status of a segment so it will treat these as uncached and add them to the list. (Fixed in v0.4)
  • Unable to get anything that is inside a BFX so all clips that have one applied will not be picked up. Upvote FI-02664 to allow that.
  • This will only grab the used version of a versioned clip, not every version available.
  • This will only grab the latest batch iteration, not previous iterations.
  • Only works on the current workspace.
  • Will not grab audio. (Fixed in v0.2)
  • Flame returns the path of an image sequence as the first frame in the sequence, not the first frame being used within said image sequence. For now the safest route is to find the last frame in the sequence on disk and use that to populate the list.

Unsure if it’s possible:

  • Access a BFX segment and get a list of clips used within it along with their attributes. (It is not, upvote feature request above)
  • See if a clip is an audio file (I didn’t stress about audio all that much…these clips were giving me errors so I’ve ignored them for now) Implemented in v0.2
  • Access other workspaces within the project. (It is not and for good reason)
  • Access clips used in previous batch group iterations and get clips used.

Who This Is For Right Now:

Someone who doesn’t use BFX (or if they do, then they have a reference to the original clip either in the timeline or someone within a library).

That said even if you’re not that person, knowing the above limitations, you do you and maybe it fills a need. Funnily enough, I personally am not the target audience but we never moved to an un-managed workflow due to this exact issue.

How To Use

Download v0.8 of the script and put it where you keep your hooks. Once you’ve refreshed your hooks you’ll see a “Collect Media” option within the main menu. Once it’s run, have a look at the shell and you’ll see where it has placed the text file as well as a rsync command to run.

By the very nature of rsync you will not have duplicated media so you can run this across however many workspaces/machines you have and be left with a clean backup.

I hope some of you find this useful. Let me know how it works out in your given situation. I’ve tried to test it fairly throughly but I’m sure there’ll be something that throws up an error somewhere.

If anyone would like to help make this more robust or to address some of the things I couldn’t figure out, please chime in. Sharing is caring as they say. @fredwarren @MikeV @claussteinmassl maybe you guys already know if any of those things are indeed possible and could point me in the right direction.

Finally, thanks to @claussteinmassl for helping me with the sub-folder issue.

Disclaimer: This isn’t bullet-proof nor is this my day job so use it at your own risk. You should always verify your backups before you remove anything.

14 Likes
2

This looks great. I’ll be trying it out.

Thanks!

3

Good effort @kyleobley

Way beyond my abilities.

4

Amazing!!!

5

Thanks all. Hopefully some of you find it useful. There’s quite a few things that would be nice to figure out so I’ll keep plugging away it.

@fredwarren Is it possible to access clips within a BFX as well as access other workspaces beyond the current one?

6

Technically it is possible to access the clips inside a BFX from the Python Console, but I would say that it is only possible to do so from the Python Console after you made some manual operations.

  1. You need to enter the BFX and this is not possible from the API itself.
  2. You need to select the clips in the Media Panel
  3. You would need to use flame.execute_shortcut() for the Exit and we do not recommend using this in scripts that you are sharing to the community.

Clip Select
When you are inside the Batch FX you can manually select the clips and then use flame.media_panel.selected_entries to get the list of selected clips but that’s pretty much it.

Enter
You would need to assign a shortcut to the “Editor…” button and then use flame.execute_shortcut("Editor…(Button)). There are two main problems with this:

  1. People who are grabbing your script would need to manually create the shortcut for the script to work.
  2. It would only work if the BFX is focused rather than any other Timeline FX that could be on the segment.

Finally, the answer is no for the Workspace. You can only access the current one.

fred

7

Thank you for the explanation. I’ll make a feature request to be able to get into BFX then.

The workspaces I totally understand since you wouldn’t want someone else being able to change things whilst you’re in it.

8

Hey @kyleobley,

please excuse that I couldn’t reply any sooner! Really great work! Thanks for all your effort, including the very detailed description! :slight_smile:

Unfortunately I cannot run and test your script as it’s only written for Python 3 and I’m still running 2020.3 at the machine I’m currently using. I don’t know, if you want to make it compatible with older versions, too. If not, it might be a good idea to specify a minimum version when adding the menu entry.
Some things could probably be adapted without much effort (print function, subprocess etc.), others need a different approach (glob module) as some functions aren’t available in older versions. So without the current ability to test the script, here are just some thoughts from reading the code:

  1. Calling a subprocess at the end to sort the list and remove duplicates:
    I think there is no need to call a subprocess with external tools. set() and sort() can do the same for your natively in python. However I think it might be even better to make sure there are no duplicates in the list while building it. As this step happens in many places in your script, why not make the lists a custom class. That way you do not need to change much in your existing code. Here is a quick example:
# python 3:
try:
	from collections import UserList
# python 2:
except:
	from UserList import UserList


class UniqueList(UserList):
    """Custom list class that acts like a set to make each item unique."""
    def __init__(self, items=None):
        self.data = []

        if items:
            if isinstance(items, UniqueList):
                self.data = items.data.copy()
            else:
                for item in items:
                    self.append(item)


    def append(self, item):
        if not item in self.data:
            super(UniqueList, self).append(item)


# before:
clips = []
sequences = []

# after:
clips = UniqueList()
sequences = UniqueList()

# test:
test = UniqueList([1, 2, 2, 3])
print test

test.append(2)
print test
  1. Docstrings & Comments:
    Docstrings (those comments starting with “”") should really only be used as such right after the declaration of a function as they are returned when running help(). For any normal comments only the hashtag should be used. :slight_smile:

.
Regarding audio:
You can access the audio file path like this:

for track in clip.audio_tracks:
    for channel in item.channels:
        for segment in channel.segments:
            print segment.file_path

.
Here again a unique list can be very handy, as a stereo-clip would also return the same file path two times with this loop. :slight_smile:

2 Likes
9

Hey Clauss,

Fair enough, I didn’t think about making it backwards compatible! Even more thanks for taking the time to write that class. These are all the Python-ness things that are cool to learn about. I suppose my logic was why re-invent something if linux has a built-in, super fast way to do the same thing. But I dig the Pythonic way of thinking so I’ll give this a go. I’ll also look more into docstrings and how all this stuff is supposed to be used. If I recall you were the one pushing the standards approach :wink:

Again, thank you for this. I’ll give it a go once I have a moment.

EDIT: Now I remember why I also wanted to use sort…this way the file could be written to multiple times and still only keep uniques but I could just read the file if it already exists and put that into the lists to start.

10

Yes, that’s true. I know this is a rather dry subject, but I really think it’s worth it. :smiley:

If you wanted, you could just read all lines from the text-file as a UniqueList, too. Merging this with the newly collected clips would also not create any duplicates:

list1 = UniqueList([1, 2, 3])
list2 = UniqueList([1, 2, 3, 4])

merged = list1 + list2
print merged
# Result: [1, 2, 3, 4]

For sorting it’s just:

# sorts the list in place (probably more useful in this case)
clips.sort()
# alternatively create a new sorted list and keep the unsorted list if needed
sorted_clips = sorted(clips)

I think doing stuff natively should be the way to go unless there is a real need for external solutions (e.g. major speed improvements, components not natively available etc.). That way you can be sure the script will just run everywhere, independent of the operating system etc. :slight_smile:

1 Like
11

Update v0.2

Download Here

Thanks to @claussteinmassl and his guidance I’ve updated the script so it can be used in versions below 2022…at least I think so. It would be great if people can test and let me know if it’s working for them.

Beyond how lists are managed, the big addition is the ability to find audio files. The same caveat exists as with video, we can’t get the cached status if it’s in a sequence so audio within one will be added to the list regardless if it’s cached or not.

Hopefully now that it will work in previous versions of Flame it’ll be of more use to everyone.

3 Likes
12

Update v0.4

Download Here

I’ve finally had the opportunity to update to 2023 which came with a needed addition to the Python API to make this script actually helpful as we can now get the cached status of any segment within a sequence.

What that means is only un-cached clips (video & audio) will now be added to the list regardless of where they reside. The same caveats still exist regarding workspaces, batch iterations and BFXs however.

As always, let me know if anything comes up and I hope some of you find it useful.

6 Likes
13

amazing @kyleobley - thanks so much for sharing. Hope you and the entire crew in the 'dam are doing well!

flame on!

-tim

1 Like
14

Thanks.I would just like to suggest the possibility of being able to choose a path for the final report, or to choose a more conventional one (documents?, desktop?.. ). I find it a bit odd to save the report in an path in opt/autodesk/proyects/. But, it’s only imho.

I miss a tool in flame to collect media to do things like that. I have been thinking for some time about a tool to open xml’s, and consolidate only footage used, in local, from external hard disks. In a way, this is pretty much the same idea. I opened this thread about it. I have a few lines of code that open an xml, and generate a list of unique paths of footagel. But my poor knowledge doesn’t allow me to go any further.

It’s just something I’m saying for anyone who wants to hear and finds it interesting.

15

The intention with having the location be a known one and connected to the project is two fold:

  1. If you have multiple workspaces within a project, you’ll have to run the script multiple times so it’s easiest to point to a common, known location.
  2. It allows you to re-run the script and not have to remember where you put the file from the previous run(s).

I suppose the idea of project/status just made sense in my head. It’s easy to change if you’d like to do so. Have a look at line 343.

1 Like
16

I’ve updated the script to v0.5. There is now the option to scrape all media, regardless of cache status. I figured it could be useful to some. I’ve also added some more error handling as I ran into a weird case in a recent project.

As always, test your backups. I can’t promise this works 100%.

Enjoy.

6 Likes
17

Verson 0.6 is now out.

Changes:

  • Fixed handling of RED footage. Previously it would only grab the first file and not the whole directory tree.
  • You can now specify the location of the collected_media.txt file within the script.
5 Likes
18

Version 0.8 is now out.

Changes:

  • Massive performance increase (1000x) due to improvements in list management.

I was trying to run this on a massive job (10 min long edit with almost a 1000 different shots) which were looking at transcoded EXRs, etc and it was taking foooorever. What was taking 20 min now takes 9 seconds with the exact same result.

7 Likes
19

@kyleobley I have never used this script myself but I just wanted to offer a huge thanks for contributing something so useful to the Flame community. If our archiving regime was different we’d be using the is all the time.

1 Like
20

Thanks @AdamArcher. Funnily enough I also wasn’t the target audience but it’s now part of our workflow as we’ve moved to unmanaged after all the chat on this forum…so it was worth the time invested. Just happy if others find it useful. It’s been a fun challange.

2 Likes