Render farm tour
on June 20th, 2010, by ideasman42Today Colin and myself recorded a tour of the renderfarm, this shows how its used by artists and the systems which render the frames.
Apologies for accidentally having areas of the video masked (would have been a spoiler).
– campbell
June 20th, 2010 at 12:54 am
Cheers to the Durian team π
June 20th, 2010 at 12:57 am
I want one…let’s see…if I give up coffee for a year I might be able to afored the electricity for one month. π
June 20th, 2010 at 1:03 am
Cool! Thanks a lot for making this video. π
I would love to get my hands on the Justa Cluster! π
June 20th, 2010 at 1:09 am
Interesting π I wold like to have my own render farm if I have the budget haha! Keep it up Durian team! π
June 20th, 2010 at 1:21 am
just curious here
what’s going to happen to this PC farm after the Durian project ?
is it going to be used by other devs to work on 2.5 !
or for the GSOC project by students ect,
Thanks and happy 2.5
June 20th, 2010 at 1:30 am
Yay an update π
That room is hot xD
Just for the sake of comparison: 30Β°C = 86Β°F
Very interesting details on the farm π And very interesting “accidents” π
June 20th, 2010 at 1:45 am
@rickyblender, realize I didn’t give a very satisfactory answer to this question. To my knowledge their are no firm plans but I think Ton can best comment on this.
June 20th, 2010 at 1:45 am
Apparently you have found a way to cool down all the chopped up bodies in the boxes of the Blender e-shop.
(when Nathan reads this he’ll get it)
June 20th, 2010 at 2:12 am
When I first saw the intro video for this renderfarm, my first thoughts were, ‘there are too many motherboards and power supplies stuffed into too small a case’. Anyhow, glad to see it’s working!!
I know it would be alot of work to make a general user interface for the community, but if you did and it could be competitively priced, while still allowing the blender foundation to make a profit, it would be another source of income for the foundation.
Personally, I won’t pay the prices some commercial renderfarms charge, so I built my own…
Randy
June 20th, 2010 at 3:07 am
Awesome tour! I had been really interested in the rendering aspect of this project and this was really informative.
June 20th, 2010 at 3:25 am
Great video, but you’re killing us with those annoying kinds of censorship! π
Well, basically it’s for our own sake in order to keep our curiosity under control… we’ll have to live with it! π
Luckily the video has just been downloaded to my laptop, so everything accidently revealed can’t be taken away from me! π
Enough silly humor, I just wanna say that you’re doing a superb job and I also would like to congratulate you for the support of Dell!
Btw, your advanced badroom airconditioning system is really impressive!
So now you’re not able to take a shower when getting sweaty on a stressful day? π
Keep on the cool work! π
June 20th, 2010 at 3:30 am
lol ideasman42 reminds me of some sort of mad scientist. You guys are awesome. haha
June 20th, 2010 at 3:41 am
YAY the Grindhouse poster!! π
June 20th, 2010 at 3:50 am
@Xero: Rather mad and useful than ”normal” and noob, isn’t it? π
June 20th, 2010 at 4:32 am
Hey, so i’m a fan of renderfarm.fi – if you wanted to open up the renderfarm… running that boinc client would be pretty cool
June 20th, 2010 at 4:39 am
You are making us all drool π Happy rendering!
June 20th, 2010 at 5:16 am
wow….so thanks
June 20th, 2010 at 5:35 am
Bet You know,
but fore the sake of procrastination (exam time again ;-)) I will put my two cents in here.
Due to its thin nature hair highly tends to alias as well as produce flickering shadows until you increase the shadow map size to an insanly high amount.
So best practice here is to make a special version of the hair with less but much thicker strands to use in your shadow pass.That way you can render fast, stable and dynamic shadows and even use your already computed hair simulation. When comping you will want to add a small amount of gaussian blur to the shadow layer, because thicker hair strands produce sharper edges on the shadows.
June 20th, 2010 at 6:59 am
Very informative… Now I have a clue of about 10% of what I dont have a clue about XD
Seriously, thank you for the tour, and the effort you guys are making to show us around the making of process. I hope that the dvd release includes these behind-the-scenes films on it.
To everyone on the team… many thanks!
June 20th, 2010 at 7:33 am
Ok, since no one else has mentioned it, i have to say that render farm management software of yours looks awesome!! I was depressed to hear that its unlikely it will be usable outside of Durian. I wonder if there would be any interested parties that could “generalize” it for any studio to adopt for their farm. I would do it, if i knew how to code…
June 20th, 2010 at 7:34 am
Thank you for showing us the server farm and what you could of the film … looking forward to the release of Sintel.
June 20th, 2010 at 7:59 am
Impressive… you guys have got an amazing workflow at the institute! Just look at all those files come and go though the SVN server… and the renderfarm administration through that web interface… not to mention the “finished” progress bar.
Guys, you rock. If you have the time, you should definitively try to document more of the process into the DVDs!
I’m really proud of how far the community has gone ^_^.
Thank you very much for the vid!!
PD: Maybe the python scripts are short or simple (isn’t that the good thing about python after all?)… but what makes all this beautiful is the way that it all works together… wonderful!
June 20th, 2010 at 8:50 am
That’s was very interesting! So much tech involved, impressive you guys can manage all that besides the movie it self.
June 20th, 2010 at 9:14 am
Thanks for another video, Colin and Campbell. π
Seeing a showerhead so close to some twenty-odd CPUs makes me queasy.
June 20th, 2010 at 10:35 am
Would be cool if you guys published the software for the renderfarm π I already tried some other programs so I could use networkrendering at home but they didn’t work =S
I hope for you guys there isn’t someone on the team who has to stay in the room with the renderfarm 24 hours a day π 30 degrees C is pretty hot!
June 20th, 2010 at 11:12 am
Please tell me you really need compiz fusionβ¦ π
And switching off the secound screen would have also been a good spoiler, well, I see, it’s more interesting this way.
Why do you have graphical interfaces on the server?
In 8:44, why did you run gnome-terimnal from the terminal? π
Anyway, you seem to have an amazing setup there! Looking forward to play around with it (whatever you’ll release from the system) once you’re done. “Anyone can to that” might be true in some way but seeing how other people solve problems is always very interesting and helps improving your own skills π
And by the way, a very nice video.
I have full trust in your skills.
June 20th, 2010 at 11:17 am
Why are you not render using boinc?
June 20th, 2010 at 11:48 am
@Ruben, yep we have a second hair system for casting shadows only, less strands and thicker as you mention.
@Joooo,
– Colin’s one of the few users who has compiz enabled, I think he knows how to turn it off but so far it hasn’t given any trouble so he leaves it on (and because he’s a mac user who likes the bling).
– About having a graphical user interface on the server. mainly so I can use SSH to access nodes from the server, (cluster ssh), but also because I like to be able to have a number of editors open at once and keep the session running over long periods. if I knew ‘screen’ I could probably avoid this.
– I use gnome terminal from a terminal because the standard xterm cant add more tabs (guessing screen would come in handy here too, general disclaimer that I’m not a sysadmin π )
June 20th, 2010 at 12:12 pm
Personally I don’t think using this as a renderfarm for a large community would be worth the all trouble for you.
Though you don’t want the hardware just sitting there collecting dust and getting out-of-date until the next movie.
So what about renting it out to a local studio for a reasonable price until next movie? I mean physically have it in their own building and they look after it.
June 20th, 2010 at 2:11 pm
My idea is sold all the components like freak merchandising. You can win a lot of money about your inversion.
I’m a network supervisor, and I can say that it’s an awesome work. Congratulations.
Saludos
adiΓ©
PD: I hate censured images.
June 20th, 2010 at 3:08 pm
But have you ever seen a renderfarm in a bathroom in your entire career?;)
June 20th, 2010 at 3:53 pm
Hi,
cool setup, can’t wait to play with it, if possible. Will there a project for a renderfarm also usable for other projects (Netrender)? I’m very interested in this.
btw: You could use LXDE/LXTerminal (Has Tabs) it’s much faster than gnome and should be enough for a renderfarm.
maces
June 20th, 2010 at 4:42 pm
All your bases are being rendered!
You are on the way to publication, make your time!
June 20th, 2010 at 5:16 pm
Question: Can we get the progress bar here on the site? Would be very cool!
June 20th, 2010 at 5:23 pm
Cool video guys! π
The plans for the renderfarm for after we’ve rendered the 2k version:
– render out the 4k version, might easily be 2 months
– optionally – if we can get budget for artists helping to make it – do the stereoscopic renders, might be 2 months as well
Apart from this, we might donate render cycles to cool & high profile projects that go on with Blender. That will be defined on a per-case basis. Think of Tube or Omega.
Obviously, we like to do a new project in 2011 as well. I’d love to find out once how much of a difference it would be to have such render power available starting day one.
Laters,
-Ton-
June 20th, 2010 at 5:43 pm
with such power available you can save a lot of time for rendering
but i guess it can also help with debuging new features in 2.5
like some new render engines in 2.5 or 2.6
but from another point of view you should consider installing some good air conditioning unit to cool off theses PC
it would extend the life expectancy of all the electronic equipment and make peoples cool or cold !
whish best of luck with this awsome set up
happy 2.5
June 20th, 2010 at 8:11 pm
For the future. What are the prospects of using LuxRender GPU rendering?
June 20th, 2010 at 8:13 pm
Greetings from Colombia
June 20th, 2010 at 8:53 pm
Having a renderfarm from day one is a blessing for artists, specialy having Γ Bucket system. Having an image every 10 minutes in design fase means 6 tries per hour, while having 1 every minute means having 60 changes made on a single design. It really helps.
The problem starts when the real renders need to be made, artists grab all the resources they can get, and images that took 5 minutes to render on 6 machines now suddenly cost 30 on 1 machine… π and you know there are artists who dont mind a 30 min wait on 6 machines. But, machines are only machines. I know I found it a blessing to no longer have to wait for minutes after each tweak.
June 20th, 2010 at 9:18 pm
cool. one day maybe i can get my own justa clusta to render my own movie. ive got interesting movie ideas.
June 20th, 2010 at 10:09 pm
very cool video! π
the cooling system of just-a-cluster doesn’t seem to be very well-thought-out though if 2 drawers always have to be open and the blender logo top has to be removed. :p how would you redesign it now with hindsight? is there some other ikea metal cabinet that would be more suited for it? or maybe the air conditioner could be directly connected to it somehow instead of just standing next to it?
1000$ per moths power bill… :s
June 20th, 2010 at 10:35 pm
I love that render farm tour… standard air conditioners,, pipe with duct tape,, a shower… it is soo charming π Could be in everyones bathroom…
June 20th, 2010 at 11:01 pm
@horace
You’ve got to know that in that small IKEA case there are 16! mainboards + power supplies running 24/7, so it’s not too bad IMHO. π (there is already a large fan in the bottom of the case.)
Maybe a suggestion:
Put an extra fan under the 2 open drawers, so you get a real straight airflow, just like IN the case.
June 21st, 2010 at 1:10 am
Lmao.. did anyone else notice the ”all your base” @ 2;47 xD
funneh ^^
June 21st, 2010 at 8:25 am
Cool too see the renderfarm getting hot!
Why doesn’t anyone ever use FarmerJoe (http://farmerjoe.info/?p=1). Now and then there is a new renderfarm script popping up while fj does a good job. I this case I see a lot shared functionality so why do you guys wrote it yourself?
June 21st, 2010 at 9:10 am
Will the 4k and stereo versions be done for a paying client? i.e someone wants a technology demo and will pay the electricity bill plus some extra for the render??
It would be nice to make some profit for the foundation to put toward a second just-a-cluster to speed along the next project…
June 21st, 2010 at 9:52 am
Hey, do you guys use Blender for all of the video and audio editing, and transitions and such? Or do you guys use a different Ubuntu program for the editing.
June 21st, 2010 at 10:14 am
WOW! it’s cool! Good work, I wish to continue in the same spirit that work!
June 21st, 2010 at 11:32 am
Nice renderfarm! It would be cool if you could publish a progressbar on this blog as J. asked.
I’ve got a question it’s about the rendertime. How well does the Blender’s rendertime compared to commercial solutions as Maya and 3D Studio Max? Let say if I want to make a short movie like 1 minute with the same quality as Sintel, would it also take several days to render on my single computer? Isn’t the rendertime for sintel like 20 days??? – if we assume it the the results becomes 27 days…
17000 frames / (24 frames/seconds) / (60 seconds/minute) = 11.8 minutes
1 minute / (11.8 minutes) * 20 days * (16 motherboards) / (1 motherboard) = 27 days… π
How much effort will the Blender development put on rendertime optimization in future releases?
June 21st, 2010 at 12:31 pm
Not sure if they are using the render branch, but you can you use the latest Trunk version of Blender from GraphicAll website and also download the Render branch and see the difference.
I benchmark (in the loosest sense of the word) a file which takes 34sec to render on the trunk version of blender vs the Render branch with taks about 7 secs….
regards
June 21st, 2010 at 12:36 pm
GIven that LuxRender is looking a being able to render on CPU/GPU/Network is there still a need for the internal render. Could Luxrender/Yafaray and the Internal render groups come out with one kick ass Renderer?
So are there going to be any out-take shots in the final release?
June 21st, 2010 at 12:53 pm
They have more than 16 motherboards, namely the 6 Dell servers, plus they also render on workstations in their ”spare time”. (a.k.a. artist not sitting at his desk)
So when everyone is to bed they have near 30 computers rattling and humming!
June 21st, 2010 at 5:53 pm
I second the suggestion of releasing the render management software for the community. I’d love to look into the code and maybe try to generalize the application for more common uses… π
Best wishes!
June 21st, 2010 at 6:40 pm
It will get released, (I heard it myself in the video)
but it’s not a bad idea to wait a moment and first let them tweak/finetune everything in their environment, let them write a manual, organize all the data etc. etc.
Open source should not be ”grab some files, put them on the internet, find out everything by yourself, well…. enjoy… if you can…
”Haastige spoed is zelden goed” (Hurry is seldom good) do we say in Holland! π
June 21st, 2010 at 7:18 pm
I would really recommend using the netrender script that is now in 2.5. We decided to use our own scripts because they were already working and easier to fit into svn and file management we had in place here, switching to a very different system was a bit risky at the time we got the renderfarm. If you want something that works out of the box, netrender works great, it also has a web interface to manage things. I’ve tested it on the render nodes here and it worked, so it would definitely be good to use that on future projects.
June 21st, 2010 at 7:31 pm
Provided that SVN gets implemented in it, right?
June 21st, 2010 at 9:53 pm
@rickyblender, rendering so many images helps a lot with quickly finding render problems but we have enough on our hand with blenders rendering engine.
We’re aware the air conditioning is pretty make-shift :S
@ajedros, as fair as I know we dont have plans to do GPU rendering at the moment at least.
@horace, in retrospect we should probably have had a stronger fan at the base of the justa-cluster, while not ideal, pulling some draws out works for now.
@Jogai. I didnt use farmer joe, but one of the reasons we have our own farm is so we can quickly integrate new features – like svn integration, rendering AVI’s from EXR files, scanning metadata from EXR’s and showing average rendertimes etc.
@Homeworkbad, this video was edited in blender, for the blog people edit video with whatever they like, we’re not as strict about being totally opensource when it comes to making content for the blog.
@patrox, I have asbolutely no idea about how blender compares to other software. One would need to run side by side tests. However brecht has been working on optimizations, some of the scenes now have half overall render times :D. Future plans havnt been made yet AFAIK.
@The Fatsnacker, yes we use the render branch sometime after durian this branch will be merged into 2.5x.
@J. Only 5 of the studio workstations are powerful enough to be used as render nodes. animators have systems with 4gig of ram and only 2 core CPU’s so they are too slow compared to quad core i7’s. (so a total of 27 nodes max)
netrender/svn integration is possible but would be more appropriate if netrender was already well tested in production. testing netrender AND adding new features would loose too much of our time.
June 22nd, 2010 at 12:57 am
That render farm is hot! Love the little GUI interface you have.
June 22nd, 2010 at 1:24 pm
ideasman Campbel said:
“However brecht has been working on optimizations, some of the scenes now have double overall render times”.
ouh shit… probably another optimization will help to get back to half the render time, maybe even less?
π
Or are here any Language issues ?
edit from a durian guy: thanks for noticing, fixed! π
June 22nd, 2010 at 1:37 pm
Not wanting to belittle your efforts but here is a video from the BBC news site (hope you guys can see it) which amongst the fluff about Toy Story 3 has a look at the Pixar Renderfarm, something to aspire to π
By the way, regarding the shower by the render farm, I think you guys may have misunderstood liquid cooling π
June 22nd, 2010 at 1:38 pm
D’Oh – forgot the link
http://news.bbc.co.uk/1/hi/entertainment_and_arts/10308227.stm
June 22nd, 2010 at 5:54 pm
@matt:
Holly crap!!! It sounds like surfing on a tsunami π It’s even intimidating.
One question:
Quote: “we might donate render cycles to cool & high profile projects that go on with Blender”
This render cycles donations would have to be done in house, like one would have to send/take the movie files physically to the blender institute or it would be possible to submit the renderings jobs from the Internet?
June 23rd, 2010 at 12:53 am
Great video! Especially for us TDs out there. Colin, I’m curious as to what codec the AVIs are compressed with after the renders are done. Are you using ffmpeg? Also, are the scenes rendered in layers? If so, do you have some sort of script that composites the rendered sequences into an EXR? Maybe this is a question more for Tom.
June 23rd, 2010 at 2:06 am
Tom? You mean Ton! π
June 23rd, 2010 at 11:50 am
Very informative indeed. For still images I have really come to love Lux render. They have a network rendering integration that is very cool and easy out of the box. Not something for Sintel, that’s for sure, but anyone else should have a look.
Back on topic: keep up the cool work!
June 23rd, 2010 at 5:36 pm
@matt: good link about pixars renderfarm, it gave a perspective on the rendertime – a complex scene in Toy Story 3 takes like 7-14 hours to render… (can’t blender have a neon-sign for its renderfarm too?;) )
June 23rd, 2010 at 6:11 pm
Hey Ideasman,
I’m running DrQueue now on a Linux machine, it’s OK, functional, but buggy as you’ve mentioned before. They also have a few web interfaces, but Blender’s interface looks much more promising. Sweet. I’m looking forward to making use of your farm tools, on my system.
I have a question, how does the server-slave communication happen? I’m curious how the server communicates with each node. Is it through SSH command line calls?
Loved the video π Thanks
June 24th, 2010 at 1:49 am
Hi, which render manager do you use for the render queue???
June 24th, 2010 at 12:20 pm
Thanks this was an awesome behind the scenes look, i think the render farm software would be useful for students from network admin backgrounds to get their feet wet.
June 24th, 2010 at 2:26 pm
π Just dreaming of my own cluster now.. Thanks guys π
June 25th, 2010 at 4:53 pm
Wow… I have no idea what most of that meant… Any possibility that we could get a tutorial about renderfarms? (After this is finished, of course) I’ve got several computers, but the only reason I can use them is that LuxRender is awesome and easy. Most of us are modelers/animators, not sysadmin, so anything that you guys could teach us would be really awesome!
Keep up the great work.
June 25th, 2010 at 6:29 pm
this render farm seems to be a very interesting subject indeed
it would be interesting to have a special page for this system of render farm
including a general description for
software use
hardware system included
how many processsor and type speed
memory in GB & hard Disk GB
and give an idea of overall peformance for that system
from a rendering point of view
and if possible give an idea of the electrical power consumption
per machine and total power
i mean $1000.00 / month seems to be a huge electrical bill
don’t know what your local cost is for Kw/ hour?
and add a link for a small video tour of the render farm too!
compare that to the power of PC over the last few decades
and it’s pretty amazing to see where we are at and how fast it is evolving over time – hardware and software wise
and it’s gone get even better and faster over next few years
happy 2.5 with high speed rendering
June 25th, 2010 at 10:03 pm
@rickyblender: Don’t forget the power consumption of the 2 airconditioning systems…
June 27th, 2010 at 12:50 pm
Thank you for video tour
June 30th, 2010 at 12:33 pm
Well, I would love to see some of the renderfarm scripts you guys have built up. I’m currently running a drqueue setup and it is slow. As a bonus I might be able to lend some of the CPU’s to speed up your 4K render — I have 1 farm running with about 220 cores and in the next little bit will hopefully be bringing on another 2000 cores π
I run the render farm for student projects, but it only gets used during class time is idle the rest of the time.
PS. love the renderfarm in a bathroom
July 1st, 2010 at 5:35 pm
@maddes, when the code isnt as well tested generally you can find bottlenecks and make bigger improvements but at some point it becomes a lot harder to improve render speed.
@BlenderBoy, we use avi-jpeg mainly because its very fast to seek and we dont get codec problems, the ffmpeg integration in blender isnt that great still π
@roydude & artramirez3D, there is a fairly simple python script which loops on a timer and sends commands to systems via ssh.
@chris, will release with the production files.
@Matt Hurley, saddly no, setting up a render farm like this still is fairly involved however you can use netrender which comes with blender 2.5, as saied before it doesnt have the features we needed but these were mostly durian spesific.
@rickyblender, this info would definetly be useful but with such a small team its hard to find time to pay attention to these details when the artists find bugs on a daily basis π
@Brent, wow, thats a massive cluster!, so I expect you know what your doing :), before durian I read up on Condor – http://www.cs.wisc.edu/condor/ its been used with blender before, looks like it could be used as a more generic way to distribute jobs.
July 2nd, 2010 at 8:42 am
Very Impressive I am using farmerjoe -the render farmer.
Working perfect on Ubuntu workstations.
I am also reading good things about corefarm.
there is now also a commercial renderfarm for Yafray.
Is there anyone familiar with corefarm?
http://www.corefarm.com/
July 2nd, 2010 at 8:59 am
sorry forget what I wrote about commercial corefarm it is online rendering
August 23rd, 2010 at 8:54 am
Iβve got a question itβs about the rendertime. How well does the Blenderβs rendertime compared to commercial solutions as Maya and 3D Studio Max? if I want to make a short movie like 1 minute with the same quality as Sintel, would it also take several days to render on my single computer? Isnβt the rendertime for sintel like 20 days??? β if we assume it the the results becomes 27 daysβ¦
17000 frames / (24 frames/seconds) / (60 seconds/minute) = 11.8 minutes
1 minute / (11.8 minutes) * 20 days * (16 motherboards) / (1 motherboard) = 27 days
How much effort will the Blender development put on rendertime optimization in future releases?
i get all the information from the google search renderrocket
January 28th, 2011 at 12:33 pm
Hi,
I’ve got about 5000 .EXR files (rendered by Blender 2.56a r34481) that I’d like to convert into .PNG:s, so I can create a nice .AVI file.
Please tell me what tool you are using to extract png:s from EXR files. I tried the exrtools suite, and Photoshop, and some other tools but the result is a black image. However, the EXR-file IS properly rendered, because I can see it look OK in the Compositor using an Image input node connected to a Viewer node. But adding the EXR files in the video editor doesn’t help either. Somehow I need to batch-import them, and I would into a very happy guy if you – or someone else could give me a hint!
Thanks
BTW Your work with Sintel is simply amazing!
Cheers!
January 28th, 2011 at 2:42 pm
Heh, it wasn’t that hard. Once I added a single Image input node, and added the first .exr file – a new dropdown option appeared, where I could select “sequence”. Then I added a composite output node, and could render the output as an animation.
October 18th, 2011 at 3:21 pm
Hi,
where can I find the php(?) script shown in the video for managing the render jobs? Is it on the DVD set?
regards
Darki