Command Line Archiving

I’m trying out command line archiving. Seems pretty straightforward. Curious who’s been using it? What their thoughts are? Can it be used while working on another project? How much it slows down the box. Working on linux.

4 Likes

I’m a huge fan. At my previous staff gig, where we would often touch a bunch of jobs in a day and/or need a lot of framestore space RIGHT NOW, being able to fire up several background processes (from another machine, if need be) was a godsend.

Working from home I haven’t needed it as much, but I was just thinking how I kind of… miss it… ? Don’t ask, I am insane.

A while back someone wrote a little gui wrapper for it but then autodesk changed some stuff and it wasn’t updated and now it’s gone/broken. But this seems like something our new best friend Python would be good for, no?

In answer to your question, it CAN be used on other projects, but it will noticeably (though not unacceptably, in my opinion) slow you down.

Command line archiving automagically in the background is life changing.

Exactly. Perhaps we can get Fred interested. He never passes up a good python script.

Tom and Tom at Gunpowder set this up for a lot of people.

I actually started this very thing 2 weeks ago and I figured out how to ssh into every machine and run “flame_archive” command (in the /opt/Autodesk/io/bin") to list all the projects on the remote machine.

I’m using the python library “paramiko” and found this thread on stack overflow that helped me a lot:
python paramiko ssh - Stack Overflow

My next step was to make a gui with a list widget and be able to select by project and then launch “flame_archive” to archive to a mounted raid.

Of course, I got too busy to finish…but maybe during Thanksgiving break I’ll post what I have.

5 Likes

Wow. Sounds amazing. A GUI would make this fool proof. Command line just opens itself up for mistakes.

…anyone have a backdraft logo we can use as a splash screen?

3 Likes

Background Archive …… yes please !

Backdraft! I had forgotten about that. Wasn’t that the machine you could do a conform on?

1 Like

I got emailed a job posting from one of the big UK shops a few months ago and they were still listing knowledge of Backdraft as a nice-to-have. Producers and their spec sheets, amirite? I did run Backdraft on Octane, though, so I must be qualified to be in charge over at Framestore, right? LET’S GOOOO! :wink:

1 Like

2 Likes

@kirk I don’t know what’s more awesome, the fact that you have that picture or that you were running X11 on your PowerBook Titanium rsh’d into an sgi when you did it.

2 Likes

We just wanted to see if it would work!

I believe this was my 2nd trip to Montréal. And around the time that @panisset referred to said TiBook as a “nice toy.”*

*not meant disparagingly, he was quick to point out.

2 Likes

Old school “blue gummy” Apple menu logo FTW. Backdraft, for a brief moment renamed to “discreet archiving utility” (since it was deemed not “product name worthy”). Backdraft was a completely separate code base, including a completely different UI toolkit which it shared with Riot (the film scanner / recorder app), I think the idea was that eventually Flame would adopt it, but that never happened. Eventually became “Conform aka Smoke with fewer buttons” when it was determined that maintaining a separate code base for a low volume product didn’t make any sense (a sensible decision IMO).

We were using Conform in anger around 2010, pointing it to 100K DPX files scanned on a Spirit to a Stornext SAN, having it conform in a few seconds felt like magic.

3 Likes

Slowly making progress

5 Likes

Didn’t someone already make a gui for bg archives? I vaguely seem to remember…

@Ton ,

Don’t know if you’re referring to my last screen cap for “Backdrafty*”. It’s an internal app that I use to archive all the flames.

Here’s the current look:

Initially, I’ve been wanting to make this open source, but just haven’t had the time. Also, there’s bugs and quirks but on the whole it works, but could be better. I suppose that’s the point of open source…others can help fix all that stuff.

Methodology

I quickly realized that you need to run “flame_archive” command on the machine you want to archive on. Which basically means you need to ssh into each machine and run “flame_archive” that way. I use the “paramiko” python library. When you create a host machine, you enter the user and password once. The credentials are never stored…it’s only used to create an ssh key and that key is stored locally:

image

Since every Flame machine has the special “/hosts” autofs folder, you can use that to navigate to a central raid to write the flame archives to.

Once you entered all your machines, you can hit “List Enabled Projects” to list all the projects on each Flame. I ended up parsing the /opt/Autodesk/project/project.db file on each machine because it was faster than the stone+wire command line tool.

Once you get the list, I will sort by the “Project” column. Then I can see which machines have projects for the same show or commercial. Way better than trying to figure this out manually.

Optionally, you can multiselect those entries and hit the “Estimate Size Selected” to see how much space you need to archive a show.

Finally, you can hit the “Archive Selected” to run the “flame_archive” command on each machine via ssh. We name our flame projects with the Job code in the beginning of the name and I use this to create folders for the archives and sub folders for each machine. There’s a console window below to show you the standard output of each process.

Also, I made a “Launch Flame” button to open up Flame locally with the remote machine and project already selected. This is so you can quickly jump into that project to do archive prep. I have another Flame python script that unlinks any rendered clip (in a very hacky way),and deletes all iterations except the last one. The prep script reduces the size of the archive by 1/100th of the size. That’s another script I want to open source eventually, but I need to refine it further.

This took a lot of work and it will take a lot more to “finish” the project. I would like to use wiretap api to auto discover flames on the network…chose which version of flame to use for an archive…have multiple console outputs for each machine…and definitely enable multi threading for each remote process.

Hit me up if you want to see the code, warts and all. But, I’m not quite ready to publish it without cleaning it up and documenting it better.

Cheers,

Danny

2 Likes

Oh another reason I’m a little hesitant to release this publicly is that this could be your IT departments worst nightmare.

I guess if you’re high enough to know the passwords to ssh into your facility’s machines then you can accept the risks that this power entails.

You can use this to batch execute arbitrary ssh commands on any machine.