Durian DVD

Follow our progress!

  • on Twitter
  • on
  • Follow our commits on Twitter!

    Render farm tour

    on June 20th, 2010, by ideasman42

    Today Colin and myself recorded a tour of the renderfarm, this shows how its used by artists and the systems which render the frames.

    Ogg-theora version

    Apologies for accidentally having areas of the video masked (would have been a spoiler).

    – campbell

    82 Responses to “Render farm tour”

    1. xyz14 Says:

      Cheers to the Durian team πŸ˜€

    2. drachels Says:

      I want one…let’s see…if I give up coffee for a year I might be able to afored the electricity for one month. πŸ™‚

    3. DingTo Says:

      Cool! Thanks a lot for making this video. πŸ™‚
      I would love to get my hands on the Justa Cluster! πŸ˜›

    4. Virion Says:

      Interesting πŸ™‚ I wold like to have my own render farm if I have the budget haha! Keep it up Durian team! πŸ˜€

    5. rickyblender Says:

      just curious here

      what’s going to happen to this PC farm after the Durian project ?
      is it going to be used by other devs to work on 2.5 !
      or for the GSOC project by students ect,

      Thanks and happy 2.5

    6. kram1032 Says:

      Yay an update πŸ˜€

      That room is hot xD
      Just for the sake of comparison: 30Β°C = 86Β°F

      Very interesting details on the farm πŸ™‚ And very interesting “accidents” πŸ˜›

    7. ideasman42 Says:

      @rickyblender, realize I didn’t give a very satisfactory answer to this question. To my knowledge their are no firm plans but I think Ton can best comment on this.

    8. Christiaan (AniCator) Says:

      Apparently you have found a way to cool down all the chopped up bodies in the boxes of the Blender e-shop.
      (when Nathan reads this he’ll get it)

    9. Revolt_Randy Says:

      When I first saw the intro video for this renderfarm, my first thoughts were, ‘there are too many motherboards and power supplies stuffed into too small a case’. Anyhow, glad to see it’s working!!

      I know it would be alot of work to make a general user interface for the community, but if you did and it could be competitively priced, while still allowing the blender foundation to make a profit, it would be another source of income for the foundation.

      Personally, I won’t pay the prices some commercial renderfarms charge, so I built my own…


    10. DwarvenFury Says:

      Awesome tour! I had been really interested in the rendering aspect of this project and this was really informative.

    11. J. Says:

      Great video, but you’re killing us with those annoying kinds of censorship! πŸ˜€
      Well, basically it’s for our own sake in order to keep our curiosity under control… we’ll have to live with it! πŸ˜‰

      Luckily the video has just been downloaded to my laptop, so everything accidently revealed can’t be taken away from me! πŸ˜‰

      Enough silly humor, I just wanna say that you’re doing a superb job and I also would like to congratulate you for the support of Dell!

      Btw, your advanced badroom airconditioning system is really impressive!
      So now you’re not able to take a shower when getting sweaty on a stressful day? πŸ˜‰

      Keep on the cool work! πŸ˜€

    12. Xero Says:

      lol ideasman42 reminds me of some sort of mad scientist. You guys are awesome. haha

    13. Blitz Says:

      YAY the Grindhouse poster!! πŸ™‚

    14. J. Says:

      @Xero: Rather mad and useful than ”normal” and noob, isn’t it? πŸ˜‰

    15. Istvan Says:

      Hey, so i’m a fan of – if you wanted to open up the renderfarm… running that boinc client would be pretty cool

    16. Robbie Losee Says:

      You are making us all drool πŸ˜‰ Happy rendering!

    17. Domy_graphy Says:

      wow….so thanks

    18. Ruben Says:

      Bet You know,
      but fore the sake of procrastination (exam time again ;-)) I will put my two cents in here.
      Due to its thin nature hair highly tends to alias as well as produce flickering shadows until you increase the shadow map size to an insanly high amount.
      So best practice here is to make a special version of the hair with less but much thicker strands to use in your shadow pass.That way you can render fast, stable and dynamic shadows and even use your already computed hair simulation. When comping you will want to add a small amount of gaussian blur to the shadow layer, because thicker hair strands produce sharper edges on the shadows.

    19. Laura Lyle Says:

      Very informative… Now I have a clue of about 10% of what I dont have a clue about XD

      Seriously, thank you for the tour, and the effort you guys are making to show us around the making of process. I hope that the dvd release includes these behind-the-scenes films on it.

      To everyone on the team… many thanks!

    20. Chad Gleason Says:

      Ok, since no one else has mentioned it, i have to say that render farm management software of yours looks awesome!! I was depressed to hear that its unlikely it will be usable outside of Durian. I wonder if there would be any interested parties that could “generalize” it for any studio to adopt for their farm. I would do it, if i knew how to code…

    21. jerms Says:

      Thank you for showing us the server farm and what you could of the film … looking forward to the release of Sintel.

    22. Federico CΓ‘ceres Says:

      Impressive… you guys have got an amazing workflow at the institute! Just look at all those files come and go though the SVN server… and the renderfarm administration through that web interface… not to mention the “finished” progress bar.

      Guys, you rock. If you have the time, you should definitively try to document more of the process into the DVDs!

      I’m really proud of how far the community has gone ^_^.

      Thank you very much for the vid!!

      PD: Maybe the python scripts are short or simple (isn’t that the good thing about python after all?)… but what makes all this beautiful is the way that it all works together… wonderful!

    23. Linkeltje Says:

      That’s was very interesting! So much tech involved, impressive you guys can manage all that besides the movie it self.

    24. D Says:

      Thanks for another video, Colin and Campbell. πŸ˜€

      Seeing a showerhead so close to some twenty-odd CPUs makes me queasy.

    25. iCandy 3D Says:

      Would be cool if you guys published the software for the renderfarm πŸ˜› I already tried some other programs so I could use networkrendering at home but they didn’t work =S

      I hope for you guys there isn’t someone on the team who has to stay in the room with the renderfarm 24 hours a day πŸ˜› 30 degrees C is pretty hot!

    26. Joooo Says:

      Please tell me you really need compiz fusion… πŸ˜‰
      And switching off the secound screen would have also been a good spoiler, well, I see, it’s more interesting this way.
      Why do you have graphical interfaces on the server?
      In 8:44, why did you run gnome-terimnal from the terminal? πŸ˜›
      Anyway, you seem to have an amazing setup there! Looking forward to play around with it (whatever you’ll release from the system) once you’re done. “Anyone can to that” might be true in some way but seeing how other people solve problems is always very interesting and helps improving your own skills πŸ™‚
      And by the way, a very nice video.
      I have full trust in your skills.

    27. comon Says:

      Why are you not render using boinc?

    28. ideasman42 Says:

      @Ruben, yep we have a second hair system for casting shadows only, less strands and thicker as you mention.

      – Colin’s one of the few users who has compiz enabled, I think he knows how to turn it off but so far it hasn’t given any trouble so he leaves it on (and because he’s a mac user who likes the bling).
      – About having a graphical user interface on the server. mainly so I can use SSH to access nodes from the server, (cluster ssh), but also because I like to be able to have a number of editors open at once and keep the session running over long periods. if I knew ‘screen’ I could probably avoid this.
      – I use gnome terminal from a terminal because the standard xterm cant add more tabs (guessing screen would come in handy here too, general disclaimer that I’m not a sysadmin πŸ™‚ )

    29. Rhys Says:

      Personally I don’t think using this as a renderfarm for a large community would be worth the all trouble for you.

      Though you don’t want the hardware just sitting there collecting dust and getting out-of-date until the next movie.
      So what about renting it out to a local studio for a reasonable price until next movie? I mean physically have it in their own building and they look after it.

    30. chotio Says:

      My idea is sold all the components like freak merchandising. You can win a lot of money about your inversion.

      I’m a network supervisor, and I can say that it’s an awesome work. Congratulations.


      PD: I hate censured images.

    31. J. Says:

      But have you ever seen a renderfarm in a bathroom in your entire career?;)

    32. maces Says:


      cool setup, can’t wait to play with it, if possible. Will there a project for a renderfarm also usable for other projects (Netrender)? I’m very interested in this.

      btw: You could use LXDE/LXTerminal (Has Tabs) it’s much faster than gnome and should be enough for a renderfarm.


    33. Riboshom Says:

      All your bases are being rendered!
      You are on the way to publication, make your time!

    34. J. Says:

      Question: Can we get the progress bar here on the site? Would be very cool!

    35. Ton Says:

      Cool video guys! πŸ™‚

      The plans for the renderfarm for after we’ve rendered the 2k version:

      – render out the 4k version, might easily be 2 months
      – optionally – if we can get budget for artists helping to make it – do the stereoscopic renders, might be 2 months as well

      Apart from this, we might donate render cycles to cool & high profile projects that go on with Blender. That will be defined on a per-case basis. Think of Tube or Omega.

      Obviously, we like to do a new project in 2011 as well. I’d love to find out once how much of a difference it would be to have such render power available starting day one.



    36. rickyblender Says:

      with such power available you can save a lot of time for rendering
      but i guess it can also help with debuging new features in 2.5
      like some new render engines in 2.5 or 2.6

      but from another point of view you should consider installing some good air conditioning unit to cool off theses PC
      it would extend the life expectancy of all the electronic equipment and make peoples cool or cold !

      whish best of luck with this awsome set up

      happy 2.5

    37. ajedros Says:

      For the future. What are the prospects of using LuxRender GPU rendering?

    38. ajedros Says:

      Greetings from Colombia

    39. Joeri67 Says:

      Having a renderfarm from day one is a blessing for artists, specialy having Γ  Bucket system. Having an image every 10 minutes in design fase means 6 tries per hour, while having 1 every minute means having 60 changes made on a single design. It really helps.
      The problem starts when the real renders need to be made, artists grab all the resources they can get, and images that took 5 minutes to render on 6 machines now suddenly cost 30 on 1 machine… πŸ™‚ and you know there are artists who dont mind a 30 min wait on 6 machines. But, machines are only machines. I know I found it a blessing to no longer have to wait for minutes after each tweak.

    40. nawabz Says:

      cool. one day maybe i can get my own justa clusta to render my own movie. ive got interesting movie ideas.

    41. horace Says:

      very cool video! πŸ™‚

      the cooling system of just-a-cluster doesn’t seem to be very well-thought-out though if 2 drawers always have to be open and the blender logo top has to be removed. :p how would you redesign it now with hindsight? is there some other ikea metal cabinet that would be more suited for it? or maybe the air conditioner could be directly connected to it somehow instead of just standing next to it?

      1000$ per moths power bill… :s

    42. Carsten Says:

      I love that render farm tour… standard air conditioners,, pipe with duct tape,, a shower… it is soo charming πŸ˜€ Could be in everyones bathroom…

    43. J. Says:

      You’ve got to know that in that small IKEA case there are 16! mainboards + power supplies running 24/7, so it’s not too bad IMHO. πŸ˜‰ (there is already a large fan in the bottom of the case.)

      Maybe a suggestion:
      Put an extra fan under the 2 open drawers, so you get a real straight airflow, just like IN the case.

    44. Ian Says:

      Lmao.. did anyone else notice the ”all your base” @ 2;47 xD
      funneh ^^

    45. Jogai Says:

      Cool too see the renderfarm getting hot!
      Why doesn’t anyone ever use FarmerJoe ( Now and then there is a new renderfarm script popping up while fj does a good job. I this case I see a lot shared functionality so why do you guys wrote it yourself?

    46. Big Fan Says:

      Will the 4k and stereo versions be done for a paying client? i.e someone wants a technology demo and will pay the electricity bill plus some extra for the render??
      It would be nice to make some profit for the foundation to put toward a second just-a-cluster to speed along the next project…

    47. Homeworkbad Says:

      Hey, do you guys use Blender for all of the video and audio editing, and transitions and such? Or do you guys use a different Ubuntu program for the editing.

    48. Π Π°Π·Π΄Π°Ρ‡Π° листовок Says:

      WOW! it’s cool! Good work, I wish to continue in the same spirit that work!

    49. patrox Says:

      Nice renderfarm! It would be cool if you could publish a progressbar on this blog as J. asked.

      I’ve got a question it’s about the rendertime. How well does the Blender’s rendertime compared to commercial solutions as Maya and 3D Studio Max? Let say if I want to make a short movie like 1 minute with the same quality as Sintel, would it also take several days to render on my single computer? Isn’t the rendertime for sintel like 20 days??? – if we assume it the the results becomes 27 days…

      17000 frames / (24 frames/seconds) / (60 seconds/minute) = 11.8 minutes

      1 minute / (11.8 minutes) * 20 days * (16 motherboards) / (1 motherboard) = 27 days… πŸ™

      How much effort will the Blender development put on rendertime optimization in future releases?

    50. The Fatsnacker Says:

      Not sure if they are using the render branch, but you can you use the latest Trunk version of Blender from GraphicAll website and also download the Render branch and see the difference.

      I benchmark (in the loosest sense of the word) a file which takes 34sec to render on the trunk version of blender vs the Render branch with taks about 7 secs….


    51. The Fatsnacker Says:

      GIven that LuxRender is looking a being able to render on CPU/GPU/Network is there still a need for the internal render. Could Luxrender/Yafaray and the Internal render groups come out with one kick ass Renderer?

      So are there going to be any out-take shots in the final release?

    52. J. Says:

      They have more than 16 motherboards, namely the 6 Dell servers, plus they also render on workstations in their ”spare time”. (a.k.a. artist not sitting at his desk)

      So when everyone is to bed they have near 30 computers rattling and humming!

    53. Tobi Says:

      I second the suggestion of releasing the render management software for the community. I’d love to look into the code and maybe try to generalize the application for more common uses… πŸ™‚

      Best wishes!

    54. J. Says:

      It will get released, (I heard it myself in the video)
      but it’s not a bad idea to wait a moment and first let them tweak/finetune everything in their environment, let them write a manual, organize all the data etc. etc.

      Open source should not be ”grab some files, put them on the internet, find out everything by yourself, well…. enjoy… if you can…

      ”Haastige spoed is zelden goed” (Hurry is seldom good) do we say in Holland! πŸ˜‰

    55. brecht Says:

      I would really recommend using the netrender script that is now in 2.5. We decided to use our own scripts because they were already working and easier to fit into svn and file management we had in place here, switching to a very different system was a bit risky at the time we got the renderfarm. If you want something that works out of the box, netrender works great, it also has a web interface to manage things. I’ve tested it on the render nodes here and it worked, so it would definitely be good to use that on future projects.

    56. J. Says:

      Provided that SVN gets implemented in it, right?

    57. ideasman42 Says:

      @rickyblender, rendering so many images helps a lot with quickly finding render problems but we have enough on our hand with blenders rendering engine.
      We’re aware the air conditioning is pretty make-shift :S

      @ajedros, as fair as I know we dont have plans to do GPU rendering at the moment at least.

      @horace, in retrospect we should probably have had a stronger fan at the base of the justa-cluster, while not ideal, pulling some draws out works for now.

      @Jogai. I didnt use farmer joe, but one of the reasons we have our own farm is so we can quickly integrate new features – like svn integration, rendering AVI’s from EXR files, scanning metadata from EXR’s and showing average rendertimes etc.

      @Homeworkbad, this video was edited in blender, for the blog people edit video with whatever they like, we’re not as strict about being totally opensource when it comes to making content for the blog.

      @patrox, I have asbolutely no idea about how blender compares to other software. One would need to run side by side tests. However brecht has been working on optimizations, some of the scenes now have half overall render times :D. Future plans havnt been made yet AFAIK.

      @The Fatsnacker, yes we use the render branch sometime after durian this branch will be merged into 2.5x.

      @J. Only 5 of the studio workstations are powerful enough to be used as render nodes. animators have systems with 4gig of ram and only 2 core CPU’s so they are too slow compared to quad core i7’s. (so a total of 27 nodes max)

      netrender/svn integration is possible but would be more appropriate if netrender was already well tested in production. testing netrender AND adding new features would loose too much of our time.

    58. Buttons Says:

      That render farm is hot! Love the little GUI interface you have.

    59. maddes Says:

      ideasman Campbel said:
      “However brecht has been working on optimizations, some of the scenes now have double overall render times”.
      ouh shit… probably another optimization will help to get back to half the render time, maybe even less?
      Or are here any Language issues ?

      edit from a durian guy: thanks for noticing, fixed! πŸ˜€

    60. matt Says:

      Not wanting to belittle your efforts but here is a video from the BBC news site (hope you guys can see it) which amongst the fluff about Toy Story 3 has a look at the Pixar Renderfarm, something to aspire to πŸ˜‰

      By the way, regarding the shower by the render farm, I think you guys may have misunderstood liquid cooling πŸ˜€

    61. matt Says:

      D’Oh – forgot the link

    62. rogper Says:

      Holly crap!!! It sounds like surfing on a tsunami πŸ˜€ It’s even intimidating.

      One question:
      Quote: “we might donate render cycles to cool & high profile projects that go on with Blender”

      This render cycles donations would have to be done in house, like one would have to send/take the movie files physically to the blender institute or it would be possible to submit the renderings jobs from the Internet?

    63. BlenderBoy Says:

      Great video! Especially for us TDs out there. Colin, I’m curious as to what codec the AVIs are compressed with after the renders are done. Are you using ffmpeg? Also, are the scenes rendered in layers? If so, do you have some sort of script that composites the rendered sequences into an EXR? Maybe this is a question more for Tom.

    64. J. Says:

      Tom? You mean Ton! πŸ˜‰

    65. LiMuBei Says:

      Very informative indeed. For still images I have really come to love Lux render. They have a network rendering integration that is very cool and easy out of the box. Not something for Sintel, that’s for sure, but anyone else should have a look.

      Back on topic: keep up the cool work!

    66. patrox Says:

      @matt: good link about pixars renderfarm, it gave a perspective on the rendertime – a complex scene in Toy Story 3 takes like 7-14 hours to render… (can’t blender have a neon-sign for its renderfarm too?;) )

    67. roydude Says:

      Hey Ideasman,

      I’m running DrQueue now on a Linux machine, it’s OK, functional, but buggy as you’ve mentioned before. They also have a few web interfaces, but Blender’s interface looks much more promising. Sweet. I’m looking forward to making use of your farm tools, on my system.

      I have a question, how does the server-slave communication happen? I’m curious how the server communicates with each node. Is it through SSH command line calls?

      Loved the video πŸ™‚ Thanks

    68. artramirez3D Says:

      Hi, which render manager do you use for the render queue???

    69. chris Says:

      Thanks this was an awesome behind the scenes look, i think the render farm software would be useful for students from network admin backgrounds to get their feet wet.

    70. Grzesiek Says:

      πŸ™‚ Just dreaming of my own cluster now.. Thanks guys πŸ˜‰

    71. Matt Hurley Says:

      Wow… I have no idea what most of that meant… Any possibility that we could get a tutorial about renderfarms? (After this is finished, of course) I’ve got several computers, but the only reason I can use them is that LuxRender is awesome and easy. Most of us are modelers/animators, not sysadmin, so anything that you guys could teach us would be really awesome!

      Keep up the great work.

    72. rickyblender Says:

      this render farm seems to be a very interesting subject indeed

      it would be interesting to have a special page for this system of render farm

      including a general description for

      software use

      hardware system included
      how many processsor and type speed
      memory in GB & hard Disk GB

      and give an idea of overall peformance for that system
      from a rendering point of view

      and if possible give an idea of the electrical power consumption
      per machine and total power

      i mean $1000.00 / month seems to be a huge electrical bill
      don’t know what your local cost is for Kw/ hour?

      and add a link for a small video tour of the render farm too!

      compare that to the power of PC over the last few decades
      and it’s pretty amazing to see where we are at and how fast it is evolving over time – hardware and software wise

      and it’s gone get even better and faster over next few years

      happy 2.5 with high speed rendering

    73. J. Says:

      @rickyblender: Don’t forget the power consumption of the 2 airconditioning systems…

    74. anonymous Says:

      Thank you for video tour

    75. Brent Says:

      Well, I would love to see some of the renderfarm scripts you guys have built up. I’m currently running a drqueue setup and it is slow. As a bonus I might be able to lend some of the CPU’s to speed up your 4K render — I have 1 farm running with about 220 cores and in the next little bit will hopefully be bringing on another 2000 cores πŸ™‚
      I run the render farm for student projects, but it only gets used during class time is idle the rest of the time.

      PS. love the renderfarm in a bathroom

    76. ideasman42 Says:

      @maddes, when the code isnt as well tested generally you can find bottlenecks and make bigger improvements but at some point it becomes a lot harder to improve render speed.

      @BlenderBoy, we use avi-jpeg mainly because its very fast to seek and we dont get codec problems, the ffmpeg integration in blender isnt that great still πŸ™

      @roydude & artramirez3D, there is a fairly simple python script which loops on a timer and sends commands to systems via ssh.

      @chris, will release with the production files.

      @Matt Hurley, saddly no, setting up a render farm like this still is fairly involved however you can use netrender which comes with blender 2.5, as saied before it doesnt have the features we needed but these were mostly durian spesific.

      @rickyblender, this info would definetly be useful but with such a small team its hard to find time to pay attention to these details when the artists find bugs on a daily basis πŸ™

      @Brent, wow, thats a massive cluster!, so I expect you know what your doing :), before durian I read up on Condor – its been used with blender before, looks like it could be used as a more generic way to distribute jobs.

    77. anton Says:

      Very Impressive I am using farmerjoe -the render farmer.
      Working perfect on Ubuntu workstations.
      I am also reading good things about corefarm.

      there is now also a commercial renderfarm for Yafray.
      Is there anyone familiar with corefarm?

    78. anton Says:

      sorry forget what I wrote about commercial corefarm it is online rendering

    79. jeff hall Says:

      I’ve got a question it’s about the rendertime. How well does the Blender’s rendertime compared to commercial solutions as Maya and 3D Studio Max? if I want to make a short movie like 1 minute with the same quality as Sintel, would it also take several days to render on my single computer? Isn’t the rendertime for sintel like 20 days??? – if we assume it the the results becomes 27 days…

      17000 frames / (24 frames/seconds) / (60 seconds/minute) = 11.8 minutes

      1 minute / (11.8 minutes) * 20 days * (16 motherboards) / (1 motherboard) = 27 days

      How much effort will the Blender development put on rendertime optimization in future releases?
      i get all the information from the google search renderrocket

    80. magnux Says:


      I’ve got about 5000 .EXR files (rendered by Blender 2.56a r34481) that I’d like to convert into .PNG:s, so I can create a nice .AVI file.

      Please tell me what tool you are using to extract png:s from EXR files. I tried the exrtools suite, and Photoshop, and some other tools but the result is a black image. However, the EXR-file IS properly rendered, because I can see it look OK in the Compositor using an Image input node connected to a Viewer node. But adding the EXR files in the video editor doesn’t help either. Somehow I need to batch-import them, and I would into a very happy guy if you – or someone else could give me a hint!

      BTW Your work with Sintel is simply amazing!

    81. magnux Says:

      Heh, it wasn’t that hard. Once I added a single Image input node, and added the first .exr file – a new dropdown option appeared, where I could select “sequence”. Then I added a composite output node, and could render the output as an animation.

    82. Darki Says:


      where can I find the php(?) script shown in the video for managing the render jobs? Is it on the DVD set?