Sintel, the Durian Open Movie Project » Blog Archive » Render farm tour


Render farm tour

on June 20th, 2010, by ideasman42

Today Colin and myself recorded a tour of the renderfarm, this shows how its used by artists and the systems which render the frames.

Ogg-theora version

Apologies for accidentally having areas of the video masked (would have been a spoiler).

– campbell

82 Responses to “Render farm tour”

  1. The Fatsnacker Says:

    GIven that LuxRender is looking a being able to render on CPU/GPU/Network is there still a need for the internal render. Could Luxrender/Yafaray and the Internal render groups come out with one kick ass Renderer?

    So are there going to be any out-take shots in the final release?

  2. J. Says:

    They have more than 16 motherboards, namely the 6 Dell servers, plus they also render on workstations in their ”spare time”. (a.k.a. artist not sitting at his desk)

    So when everyone is to bed they have near 30 computers rattling and humming!

  3. Tobi Says:

    I second the suggestion of releasing the render management software for the community. I’d love to look into the code and maybe try to generalize the application for more common uses… πŸ™‚

    Best wishes!

  4. J. Says:

    It will get released, (I heard it myself in the video)
    but it’s not a bad idea to wait a moment and first let them tweak/finetune everything in their environment, let them write a manual, organize all the data etc. etc.

    Open source should not be ”grab some files, put them on the internet, find out everything by yourself, well…. enjoy… if you can…

    ”Haastige spoed is zelden goed” (Hurry is seldom good) do we say in Holland! πŸ˜‰

  5. brecht Says:

    I would really recommend using the netrender script that is now in 2.5. We decided to use our own scripts because they were already working and easier to fit into svn and file management we had in place here, switching to a very different system was a bit risky at the time we got the renderfarm. If you want something that works out of the box, netrender works great, it also has a web interface to manage things. I’ve tested it on the render nodes here and it worked, so it would definitely be good to use that on future projects.

  6. J. Says:

    Provided that SVN gets implemented in it, right?

  7. ideasman42 Says:

    @rickyblender, rendering so many images helps a lot with quickly finding render problems but we have enough on our hand with blenders rendering engine.
    We’re aware the air conditioning is pretty make-shift :S

    @ajedros, as fair as I know we dont have plans to do GPU rendering at the moment at least.

    @horace, in retrospect we should probably have had a stronger fan at the base of the justa-cluster, while not ideal, pulling some draws out works for now.

    @Jogai. I didnt use farmer joe, but one of the reasons we have our own farm is so we can quickly integrate new features – like svn integration, rendering AVI’s from EXR files, scanning metadata from EXR’s and showing average rendertimes etc.

    @Homeworkbad, this video was edited in blender, for the blog people edit video with whatever they like, we’re not as strict about being totally opensource when it comes to making content for the blog.

    @patrox, I have asbolutely no idea about how blender compares to other software. One would need to run side by side tests. However brecht has been working on optimizations, some of the scenes now have half overall render times :D. Future plans havnt been made yet AFAIK.

    @The Fatsnacker, yes we use the render branch sometime after durian this branch will be merged into 2.5x.

    @J. Only 5 of the studio workstations are powerful enough to be used as render nodes. animators have systems with 4gig of ram and only 2 core CPU’s so they are too slow compared to quad core i7’s. (so a total of 27 nodes max)

    netrender/svn integration is possible but would be more appropriate if netrender was already well tested in production. testing netrender AND adding new features would loose too much of our time.

  8. Buttons Says:

    That render farm is hot! Love the little GUI interface you have.

  9. maddes Says:

    ideasman Campbel said:
    “However brecht has been working on optimizations, some of the scenes now have double overall render times”.
    ouh shit… probably another optimization will help to get back to half the render time, maybe even less?
    Or are here any Language issues ?

    edit from a durian guy: thanks for noticing, fixed! πŸ˜€

  10. matt Says:

    Not wanting to belittle your efforts but here is a video from the BBC news site (hope you guys can see it) which amongst the fluff about Toy Story 3 has a look at the Pixar Renderfarm, something to aspire to πŸ˜‰

    By the way, regarding the shower by the render farm, I think you guys may have misunderstood liquid cooling πŸ˜€

  11. matt Says:

    D’Oh – forgot the link

  12. rogper Says:

    Holly crap!!! It sounds like surfing on a tsunami πŸ˜€ It’s even intimidating.

    One question:
    Quote: “we might donate render cycles to cool & high profile projects that go on with Blender”

    This render cycles donations would have to be done in house, like one would have to send/take the movie files physically to the blender institute or it would be possible to submit the renderings jobs from the Internet?

  13. BlenderBoy Says:

    Great video! Especially for us TDs out there. Colin, I’m curious as to what codec the AVIs are compressed with after the renders are done. Are you using ffmpeg? Also, are the scenes rendered in layers? If so, do you have some sort of script that composites the rendered sequences into an EXR? Maybe this is a question more for Tom.

  14. J. Says:

    Tom? You mean Ton! πŸ˜‰

  15. LiMuBei Says:

    Very informative indeed. For still images I have really come to love Lux render. They have a network rendering integration that is very cool and easy out of the box. Not something for Sintel, that’s for sure, but anyone else should have a look.

    Back on topic: keep up the cool work!

  16. patrox Says:

    @matt: good link about pixars renderfarm, it gave a perspective on the rendertime – a complex scene in Toy Story 3 takes like 7-14 hours to render… (can’t blender have a neon-sign for its renderfarm too?;) )

  17. roydude Says:

    Hey Ideasman,

    I’m running DrQueue now on a Linux machine, it’s OK, functional, but buggy as you’ve mentioned before. They also have a few web interfaces, but Blender’s interface looks much more promising. Sweet. I’m looking forward to making use of your farm tools, on my system.

    I have a question, how does the server-slave communication happen? I’m curious how the server communicates with each node. Is it through SSH command line calls?

    Loved the video πŸ™‚ Thanks

  18. artramirez3D Says:

    Hi, which render manager do you use for the render queue???

  19. chris Says:

    Thanks this was an awesome behind the scenes look, i think the render farm software would be useful for students from network admin backgrounds to get their feet wet.

  20. Grzesiek Says:

    πŸ™‚ Just dreaming of my own cluster now.. Thanks guys πŸ˜‰

  21. Matt Hurley Says:

    Wow… I have no idea what most of that meant… Any possibility that we could get a tutorial about renderfarms? (After this is finished, of course) I’ve got several computers, but the only reason I can use them is that LuxRender is awesome and easy. Most of us are modelers/animators, not sysadmin, so anything that you guys could teach us would be really awesome!

    Keep up the great work.

  22. rickyblender Says:

    this render farm seems to be a very interesting subject indeed

    it would be interesting to have a special page for this system of render farm

    including a general description for

    software use

    hardware system included
    how many processsor and type speed
    memory in GB & hard Disk GB

    and give an idea of overall peformance for that system
    from a rendering point of view

    and if possible give an idea of the electrical power consumption
    per machine and total power

    i mean $1000.00 / month seems to be a huge electrical bill
    don’t know what your local cost is for Kw/ hour?

    and add a link for a small video tour of the render farm too!

    compare that to the power of PC over the last few decades
    and it’s pretty amazing to see where we are at and how fast it is evolving over time – hardware and software wise

    and it’s gone get even better and faster over next few years

    happy 2.5 with high speed rendering

  23. J. Says:

    @rickyblender: Don’t forget the power consumption of the 2 airconditioning systems…

  24. anonymous Says:

    Thank you for video tour

  25. Brent Says:

    Well, I would love to see some of the renderfarm scripts you guys have built up. I’m currently running a drqueue setup and it is slow. As a bonus I might be able to lend some of the CPU’s to speed up your 4K render — I have 1 farm running with about 220 cores and in the next little bit will hopefully be bringing on another 2000 cores πŸ™‚
    I run the render farm for student projects, but it only gets used during class time is idle the rest of the time.

    PS. love the renderfarm in a bathroom

  26. ideasman42 Says:

    @maddes, when the code isnt as well tested generally you can find bottlenecks and make bigger improvements but at some point it becomes a lot harder to improve render speed.

    @BlenderBoy, we use avi-jpeg mainly because its very fast to seek and we dont get codec problems, the ffmpeg integration in blender isnt that great still πŸ™

    @roydude & artramirez3D, there is a fairly simple python script which loops on a timer and sends commands to systems via ssh.

    @chris, will release with the production files.

    @Matt Hurley, saddly no, setting up a render farm like this still is fairly involved however you can use netrender which comes with blender 2.5, as saied before it doesnt have the features we needed but these were mostly durian spesific.

    @rickyblender, this info would definetly be useful but with such a small team its hard to find time to pay attention to these details when the artists find bugs on a daily basis πŸ™

    @Brent, wow, thats a massive cluster!, so I expect you know what your doing :), before durian I read up on Condor – its been used with blender before, looks like it could be used as a more generic way to distribute jobs.

  27. anton Says:

    Very Impressive I am using farmerjoe -the render farmer.
    Working perfect on Ubuntu workstations.
    I am also reading good things about corefarm.

    there is now also a commercial renderfarm for Yafray.
    Is there anyone familiar with corefarm?

  28. anton Says:

    sorry forget what I wrote about commercial corefarm it is online rendering

  29. jeff hall Says:

    I’ve got a question it’s about the rendertime. How well does the Blender’s rendertime compared to commercial solutions as Maya and 3D Studio Max? if I want to make a short movie like 1 minute with the same quality as Sintel, would it also take several days to render on my single computer? Isn’t the rendertime for sintel like 20 days??? – if we assume it the the results becomes 27 days…

    17000 frames / (24 frames/seconds) / (60 seconds/minute) = 11.8 minutes

    1 minute / (11.8 minutes) * 20 days * (16 motherboards) / (1 motherboard) = 27 days

    How much effort will the Blender development put on rendertime optimization in future releases?
    i get all the information from the google search renderrocket

  30. magnux Says:


    I’ve got about 5000 .EXR files (rendered by Blender 2.56a r34481) that I’d like to convert into .PNG:s, so I can create a nice .AVI file.

    Please tell me what tool you are using to extract png:s from EXR files. I tried the exrtools suite, and Photoshop, and some other tools but the result is a black image. However, the EXR-file IS properly rendered, because I can see it look OK in the Compositor using an Image input node connected to a Viewer node. But adding the EXR files in the video editor doesn’t help either. Somehow I need to batch-import them, and I would into a very happy guy if you – or someone else could give me a hint!

    BTW Your work with Sintel is simply amazing!

  31. magnux Says:

    Heh, it wasn’t that hard. Once I added a single Image input node, and added the first .exr file – a new dropdown option appeared, where I could select “sequence”. Then I added a composite output node, and could render the output as an animation.

  32. Darki Says:


    where can I find the php(?) script shown in the video for managing the render jobs? Is it on the DVD set?