I've read through a lot of threads, and seem to remember seeing some that i've not been able to find again, i'm hoping i can get some pointers overall and hopefully also contribute back.
I'm using AE to process a lot of footage for a feature. My other tool sets include Nuke and Modo, but AE is useful because it's scriptable and the dark energy plug in works with it. I'm also having to use it to fix some camera hardware issues too. So it's been really good for me. However when dealing with 10,000s frames any performance tweaks can make a big difference for me
My observations/questions are:
I'm running a few different workstations and dedicated render nodes around shared storage. At the moment the shared storage is around a synology 1812+ and 513 NAS over gigabit ethernet, i can't quite run to a fibre SAN, however i am looking at a bigger NAS (the 3612xs which offers 4xGigE and 2x10GigE).
One observation is that i have my processing split into different AE projects and each project links to about 50-100 source ProRes and AVCHD files, each project takes around 10 minutes to open and i think that's really slow. The project and the source files are on the NAS. I render the results to local disc, that sped things about about 20% so the network storage is just reading. My question is what is AE doing with these source QT movies that takes so long to open, is it scanning them through or something?
One output of my pipeline will probably be 2K OpenEXR files, and i remember reading some big caveats about rendering file based with MP in that AE creates a storm of reads/writes. I haven't been able to find this thread again?
My CPU utilisation also seems quite low and my network is around 12%. In other words i'm not sure what the bottle necks are or how to go about finding them. Whether there are tools out there to anaylse network traffic - is it bandwidth? latency? I've tweaked many MP settings, so each thread has lots of ram, running 2 to 4 cpus but never managed to saturate some aspect of the processing. I just don't know what is taking the time. In this first stage files are being written as QT and i know that QT has a huge overhead in terms of writing (compared to AVI for example). But again, techniques for understanding and troubleshooting this would be useful to hear?
So any pointers would be greatfully receieved!
cheers
Paul