Developer *Fail* Moments
on July 14th, 2010, by ideasman42
Over the course of the project its come to my attention that as developers we get it pretty easy with lots of encouraging comments from the community, however I worry that we don’t properly communicate the troubles we invariably cause when poking about with Blender.
To this end, here is a laundry list of memorable mistakes we made during the project.
– Campbell 🙂
- Wondering why his IP address has a collision with the server, Brecht changed it so his system could become a contributing render node, for some reason nobody can access the server.
- Campbell spent 3 hours to find out why converting to mesh wasn’t working, copying from the wrong object, one character change.
- Campbell spent an hour or so trying to work out why the smaller the pow(size, 0.0001), then pow(size, 0.0000000001) became more accurate. Turns out no multiplication was needed the rotation matrix has the scale already applied: pow(anything, 0.0)==1.0
- After considered discussion Campbell and Brecht were sure the artists didn’t have custom patches applied and it was save to refresh their builds, Brecht modifys the build script wiping Campbell’s build directory, documentation scripts and WIP patches, Later we discovered Nathan also had 2 uncommitted rig-generating scripts, Luckily he managed use some ninja undelete-fu to get them back.
- Campbell committed removal of most of blenders source code while attempting a merge.
- Brecht’s tiling/mipmap system compares file access times which the farm ignored, dancing textures!
- Campbell updated the RNA API for keyframe paths, forgot to search for function calls and totally broke keyframe insertion.
- Ton made custom keymaps override everything else and broke grease pencil
- Campbell wrote a multi-processing hair-sim generator to run overnight which managed to overwrite all the freshly generated data while trying to render result for review.
July 14th, 2010 at 12:46 pm
hahaha
July 14th, 2010 at 1:11 pm
😀
July 14th, 2010 at 1:19 pm
laughing out loud! reminds me of so many of my mistakes, and I’m not even a dev.
July 14th, 2010 at 1:41 pm
It also reminds me some mistakes in my former work in electronics : Spending half a day on wiring a huge special plug and discovering when finished that caps and rings had not been put on the cable !
July 14th, 2010 at 1:45 pm
you know what blender needs, though? Some sort of protect cache button for sims. I really love sitting through an hour of sim-ing and then accidentally change one variable and it has to re-sim EVERYTHING.
July 14th, 2010 at 1:57 pm
This is the most depressing Durian entry ever! D:
July 14th, 2010 at 2:12 pm
Good to see you guys can laugh at yourselves sometimes it is hard to get over your own mistakes.
July 14th, 2010 at 2:17 pm
Remember Kids, when Campbell is around BACKUP your systems!
July 14th, 2010 at 2:18 pm
# D Says:
>July 14th, 2010 at 13:57
>
>This is the most depressing Durian entry ever! D:
As a developer, I’d say it’s always encouraging to see I’m not the only one to make such mistakes 😉
July 14th, 2010 at 2:22 pm
Fortunately, the team knows the rule : “Remember: Data you don’t have at least two copies of is data you don’t care about.” 🙂
July 14th, 2010 at 2:29 pm
i feel ya man. image tell more than a` thousand words.
July 14th, 2010 at 2:32 pm
I’ll be the first one to admit that the most common programming and ble(u)ndering error is usually resolved and filed under
resolution code: ID10-T
Hey that’s a good one: its a combination of Blendering and Blundering –
so I’m Ble(u)ndering!
July 14th, 2010 at 2:35 pm
i remember opening my lamp because it wasnt working, 3 hours later i needed to switch it on for it to turn on DNNNNN
:S
July 14th, 2010 at 2:51 pm
hahah These moments must be total adrenaline. :>
July 14th, 2010 at 2:57 pm
Why are there even uncommitted scripts/patches? Oh, you don’t use a distributed source control management (DSCM) system. I’d say it’s time to change that! 🙂
July 14th, 2010 at 3:02 pm
Humans? You let humans code?
July 14th, 2010 at 3:18 pm
the fault is a part of human kind
July 14th, 2010 at 3:28 pm
Developing software is like skiing… if you don’t fall over, you’re not trying hard enough!
July 14th, 2010 at 3:56 pm
And you share this with everyone 😮 ?
It’s good to know what kind of problems you face. Thanks for the update 😀
July 14th, 2010 at 4:09 pm
And after all Blender still rocks (and things get commited). Thank you guys, you are awesome!
July 14th, 2010 at 4:28 pm
Its good to see that its not just me that make such “Fail moments”. =P
July 14th, 2010 at 6:18 pm
Very funny 🙂
I would be surprised, though, if there are not at least ten times the problems mentioned here, on a project of this size…
July 14th, 2010 at 6:35 pm
Lol, I applaud you guys for sticking your necks out there like that! To err is to be human though and you and all the Durian team are doing a great job!!
July 14th, 2010 at 6:55 pm
Developer or artist, some days here can only be summed up by the following:
http://i524.photobucket.com/albums/cc326/dgiharris/DoubleFacePalm.jpg
Kudos to the devs for their work BTW. 🙂
July 14th, 2010 at 7:28 pm
Should change the title to “Get a blender developer of your own”.
July 14th, 2010 at 9:20 pm
The only people who don’t make mistakes don’t do anything (and usually consider themselves experts :))
As someone who has been programming for nearly 30 years (earning a living at it for 17+ of them) I have to say I’ve had a few double facepalm moments of my own … the trick is to learn from them and find new mistakes to make next time 😀
July 14th, 2010 at 10:01 pm
Being a Render Wrangler, folder name conventions and to which folder on the disk mount is for final or artist review. Can be hell for us when we have to QA team the final output for production quality review.
Also, the merging of svn’s *OUCH*. Doesn’t Mericurial (Google Code supports it) or GIT handle that better.
Something to consider reading.
http://arstechnica.com/open-source/news/2009/04/google-code-adds-mercurial-version-control-system.ars
Here’s the direct link to Google Code analysis, http://code.google.com/p/support/wiki/DVCSAnalysis
Be sure to read the comments as there are some chatter and difference over the history point in the comparison.
July 14th, 2010 at 10:09 pm
*correction* Also, the merging of svn’s *OUCH*. Doesn’t Mericurial (Google Code supports it) or GIT handle that better.
I was referring to the history aspect of the DVCS. Doesn’t user have the source on their local disk as well, that way they can go back a revision.
@Nathan What was the ninja undelete-fu?
July 14th, 2010 at 11:03 pm
Wow, despite all of the set backs and problems, the movie still got done! Durian Team Amazing Feat #26144!
July 14th, 2010 at 11:34 pm
oh teh joy of programming!
July 14th, 2010 at 11:38 pm
@Alexsani, once cache is external I found it doesn’t get removed, but this definitely is an area that needs work.
@MadSquirrel, surprisingly very few of our problems have been todo with data loss/backups etc. We have however had the server disk die but luckily was able to mirror it and swap it out.
@alalo, if you want to raise your heard rate try formatting a USB disk at 2am on a server where switching /dev/sdb with /dev/sdd would wipe all your art assets :D. Now I tend to do this on my workstation instead.
@panzi, as a rule we commit everything to Subversion, but when working on stuff we don’t always do this immediately. As for DVCS, Id like to try GIT but didn’t get around to it yet.
@yoff, for sure, things like this happen all the time though normally there more complicated and hard to explain- Fix X breaks feature Y in corner case Z, less facepalm moments and more “Isn’t blender complicated, perhaps we should ***remove some features!***”
@Anonymous Coward, we’re taken!
@matt, exactly, when approaching problems its good to get a sense of which things to check first to avoid loosing a lot of time on dead-end leads.
@Whimsy Collective, yep, experiencing this now, lots of files being rendered, flickering, restarting nodes checking farm heat… etc.
SVN Does merging well enough (though not as well as GIT I hear)
was totally my fault.
Thanks for the link, as said above, Id like to try DVCS at some point.
July 14th, 2010 at 11:55 pm
Entertaining in a dark kind of way. We need a pixar type out takes for sintel 🙂
July 15th, 2010 at 12:33 am
An idea: Scales getting angry and biting in Sintel’s finger. 😀
July 15th, 2010 at 3:15 am
ouch…i’ve had a few moments like that…not nearly as serious, but still, i understand…i think
but Blender is still awesome, and you devs are still awesome 😀 really big thanks for all the great work you’ve done – Blender is now even awesome-er than before 😀 GO DEVS!
July 15th, 2010 at 4:54 am
LOL! Quite honestly, this was one of the best and most entertaining posts I’ve read on this project. People don’t usually understand how frustrating development can be at times, and just how easy we can over look things or completely screw things up. Thanks for shedding some light on it by sharing your experiences. Keep up the good work.
July 15th, 2010 at 5:01 am
Oh yeah, Campbell if you do investigate version control software in the future, please experiment with Mercurial. I know everyone loves to use GIT, however, mercurial is pretty nice one too. I don’t know which one would suit Blender’s needs better but just thought I’d mention it.
July 15th, 2010 at 5:24 am
eheheheh
very funny 😀
July 15th, 2010 at 6:15 am
Link for Git Front-end tools, and what not…
https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools
Git-Cola for front-end seems by comparison among the others.
July 15th, 2010 at 9:40 am
The joys of software development! Oh man those were pretty hilarious.
On the same note, in my recent programming classes (I’m a soft. eng. student), I’ve often helped friends who were stuck on an error message for very long to notice that they’d forgotten a semi-colon at the end of a class declaration which once included elsewhere was breaking more than half their code.
I feel your pain and hope you guys are pulling through all right 🙂
July 15th, 2010 at 10:07 am
Am I the only one who thinks that at this stage of the project, there should be much more eye candy posts, even if they were just a couple of pictures or rough renders or whatever?. This kind of posts are good too, but come on:
world cup–> hire a Durian member–>Developer facepalm—>???
please, give us some eyecandy, just a little, nice, hd or even sd snapshots!!
July 15th, 2010 at 11:39 am
Lol.
Being a good programmer makes code fluid.
So yeah, fluid splatters.
Today I kept wondering why adding an object didnt work, untill I remembered I’m in object select mode and the new objects where added in the scene… Argh.
July 15th, 2010 at 12:37 pm
Nawww….Campbell needs a hug….:(
July 15th, 2010 at 2:09 pm
Lol. I’m not brave enough to blog about my mistakes, but if I was, you guys would laugh pretty loud too.
July 15th, 2010 at 3:26 pm
@ Juan Romero (Onnevan)
I completely agree with you. I like all the curent posts too, but hey, the team ought to be able to SHOW something interesting.
For example: Several months (!) ago they promised to keep us informed about the progress of the first minute – anyone remembers about that?
July 15th, 2010 at 4:25 pm
ideasman42:
I made that exact same mistake you mentioned when trying to reformat a thumb drive in a hurry! It was hell for a while! Why did they have to make the names so similar?! I thought i’d lost everything on my hard drive but then I discovered photorec. I would recommend it to anyone in a similar situation. Chances are, you’ve probably used it before!
July 15th, 2010 at 5:31 pm
LOL…..hilarious yet depressing. are you guys going to have a technical blooper reel on the DVD? that would be awsome…… ya there has been plenty of times where I have been working on a project, then suddenly you would see me screaming and tearing my hair out because one little change and a whole scene is messed up. keep up the good work guys
July 15th, 2010 at 9:47 pm
it’s nice to see, how you guys handle that. Where I’ve been working as a coder, mistakes were handled like “oh my goodness, you’re the most worthless douchebag I’ve ever met. I’ll have to report that to the boss” – when that guys made mistakes it was like “it ‘s just a crappy system… AND the fault of the project manager” 🙂
July 16th, 2010 at 1:50 am
So Ton made only *one* mistake…
Ton is impressive. 😉
July 16th, 2010 at 2:39 am
Big Fan: No this is sade! Because Ton is human then. 🙂
DurianTeam: You can even fail to finish the movie that you will remain our heros! :p
July 16th, 2010 at 2:51 am
Are you out of your mind?
Of course they’ll finish the movie, maybe with Ton in the ” angry James Cameron-like mode”, but they will! 😛 😛 😛
July 16th, 2010 at 6:29 am
those kind of cock ups are par for the course! well done on weathering them in good humour and not knocking each others lights out.(presumably)
July 16th, 2010 at 4:10 pm
This further is pushing me to learn the Blender source. Ton *did* say to me “we need coders.” and more watchful eyes, and more people to attack bugs (especially with Matt leaving full time bug squashing) certainly can’t hurt.
You should append a little bit at the end there like “Want to have just as much fun? Try these links to get started with Blender source!” lol
Secondly, to admit you’re wrong is bold. That deserves a donation. $$$
July 16th, 2010 at 5:08 pm
@ Juan Romero (Onnevan)
@ maddes
I totally agree also. Very simple posts like this
/goodies/concept-art-dragons/
or this
/production/render-gallery-update/
are just plain gold to look at!
The first minute might have been converted to more complicated issue to share if it has developed to reveal too much information – but at least more of these “dump stuff from our ‘weekly’ folders” (said in the latter link) would be more than welcome – in addition to these also great techy posts.
July 17th, 2010 at 7:20 pm
Shit happens dudes, but the work you’re doing is awesome and very much appreciated. Keep up the good work guys.
July 20th, 2010 at 3:50 pm
in spite of these failures , Blender developers team had done great job
🙂
July 20th, 2010 at 7:06 pm
LOL 🙂
It happens me all the time, at least I’m not alone 😉
Cheers and very good post, we learn as much from failures than success
July 31st, 2010 at 5:21 pm
I understand how horrible all this is, even using 2.49b it was still hell trying to manage things and keep my mind on track and not want to turn my skin inside out.