Open Source 3D Animated Series

February 19th, 2020 - Frustration

I spent the last couple of days frustrated with the flex rig file. I asked a question on blender exchange, but no one answered: https://blender.stackexchange.com/questions/167308/cg-cookie-flex-rig-linking-problems

There were 4 problems

1. Invalid drivers and cyclic dependencies
   • I was thinking these were the reason for the other issues I list below. Turn out not, but I cleaned them up. I had to write some python scripts to cipher through all the drivers and find the invalid ones. At one point I had to open the blend file in a hex editor and look for a driver that was bad but I couldn't see it through the interface.
2. The rig wasn't visible after I make proxy. It was hidden behind the linked character.
   • I describe how I solved this problem in this video
3. The bone shapes had modifiers on them, which disappear when I execute the rig_ui script (or under other mysterious circumstances)
   • I just had to apply the modifiers so they don't disappear
4. The IK leg was bending backwards. So the knee would bend backwards instead of forwards. Reminded me of the aliens in that Charlie Sheen movie (Arrival)
   • I describe how I solved this issue in the same video above.

Have to admit this behaviour is kinda weird all around. Not sure if it's because the file was originally 2.79 or what. When I use rigify with MB-Lab things were a lot smoother. Anyway, moving on.

FYI, the file can be downloaded from here

February 2nd, 2020 - CG Cookie Flex-Rig

I'm back at the technical work again. I think I settled on using the CG Cookie Flex Rig. The character is quiet descent and the rig is very nice. It needed IK/FK snapping for the arms and legs. I did some code splicing between the MB-Lab character with Rigify and pulled out the needed bits and pieces to implement IK/FK snapping on the CG Cookie Rig. That seemed to work pretty well. You can download the updated blend file here.

Here is my plan for the next while

September 26th, 2019 - Update

Been a while since I updated this blog. This is going to be a long haul project.

I finished a draft of the script. And I wanted someone to go over it. Honestly, it's hard finding someone to collaborate with. I decided to simply higher someone with writing experience to go over the script and polish it. I met one guy and we got to chatting. He had a few good questions, which motivated me to go back and re-examine the story. Sort of go back to the basics. I used the story I had already developed as basis for a new version of the story. I'm planning to spend a couple of days with that writer to fine tune it and make sure it's complete and captivating.

In the meantime, I ran into a really interesting show called, "Undone". It's been developed in a really interesting way. They call it rotoscope animation. Basically they film the actors and then paint over them, and create the rest of the elements in 3D or 2D animation. "Scanner Darkly" was done in the same way I believe. Here is a trailer

After watching that show it got me thinking about creating a new visual style for the show. After going through the animation test I've done, I came to realize the importance of real actors to get the emotions right. Especially, if I'll be using character generation tools like MB-Lab, I won't have the fine control to express the emotions. The tool just doesn't have the rig to do that.

What would be ideal is a mix of real actors and 3D animation. But doing something photo-realistic is simply beyond my means. I'm currently asking myself this question: Is there a visual style which can capture the actors emotions, but allow me to build a grand world without having a billion dollar budget?

Investigating...

August 20th, 2019 - Yet Another Update

Still working on the practice animation scene. I'm further along now. One thing I'm starting to realize with that scene is the cuts. There are some unnatural cuts that break up the action of the scene. They were done to insert the reactions of the actors. But when I'm trying to translate that into one continuous scene, the pauses feel unnatural, so I have to keep adding some actions to smooth the transitions.

July 30, 2019 - Run Cycle

To start off I decided to make a short animated scene based on: https://youtu.be/qXPOl6EjbWg?t=96. The idea is to use that as training ground for setting up my animation workflow. The first part of this shot is Captain America running towards Winter Soldier. Decomposing the work further, I decided to make a run cycle. Here is what I have. Animation is tricky. I struggled getting a descent run cycle until I used a reference run. This is the key to getting good animation. What I have probably requires more polishing, but I don't want to get bogged down with all the details. I want to have a passable animation, then refine it later once I have a scene put together. We'll see how that works.

July 11, 2019 - Gemma

Well, I got around to doing some "modelling". Between quotes because I'm using the MB-Lab blender add-on. I did quiet a few tweaks to it, specifically, with the Eye shader and the face rig. I also went back to the original Manuel Bastioni Lab skin shader. He got it right the first time. The results, to my eyes, are better than what the official MB-Lab has. Anyway, I think I have my main character, Gemma. Still needs hair, but hey, it's 2019 :)

Even though, I wasn't really modelling, per say. I had to use a reference for the character. I used her: https://www.instagram.com/mona__hala/

If this ever gets made into a live action series, I'll ask her to play the main part... haha. Anyway, my point is that using a reference gave me a target to shoot for.

June 9th, 2019 - Vancouver, I have a script

Yup! Can you believe it? I not only have one script I have an entire season, 10 episodes. Granted each episode is about 3-5 minutes long, but hey, it's a full story. The way I figure it what I wrote can be a Youtube/online season, or it can be a 45 minute pilot episode. Since I'm making it, I'll go with the former.

After months of brooding and moaning and chest beating, I sat down and wrote all 10 episodes in two days. Weird, eh?

Here comes my next dilemma. I want to share it publicly, but like any artist I have some insecurities that it'll get torn to shreds. Of course, there isn't that many people (if any) following this site, but I'm still not sure if I should share it publicly.

Here is what I'm thinking. I'll keep it under wraps for now and I'll share the output of the actual production cycle publicly, including all 3D resources, files, etc.

But I'm looking for feedback, so if you're interested in reading the scripts and providing constructive and detailed criticism ping me here.

May 4, 2019 - Where to now?

It's been a while since I updated this blog. I've hit a bit of a lull. Work has been taking all my time. I've been thinking though, where to go from here. There are two directions. I can keep going down the technical path. Try to enhance my facial expressions system. Or I can start revisiting my story. I really want to produce something creative. So I decided to head down the creative road. I'm working on a short film idea that has some potential. Once I have it well formed, I'll share it on here. I still like the direction the story I have outlined here is taking, but I'll take a break from it and explore a slightly different world.

My dad used to tell me write what you know. I never understood what that means. If I only write what I know, I'll write really boring stories. But I think I have modified his advice to something that makes more sense to me. Write the characters you know. The most important aspect of a story is the character. If you analyze any movie you like you'll come to the same conclusion as I did. Characters are what draws you into the story and keeps you glued to the screen or the pages. They are the ones who make you feel something. You relate to them, and you care what happens to them. The challenges and the "plot" is what brings out the personality of the character. They are the events which show what the character is made out of; their courage; their compassion; their fears; their love. And these aspects is what makes the viewer identify with your character.

As a writer the character, I believe, has to come from a personal place. In essence they are a part of you. If you don't write from what you know, the character you write will feel contrived and fake. But if you write from your own experiences, then they might, just might, be relatable. For this reason, I decided to write a character which I have personal experiences with. The world I'm thinking of is not a real world. It's a futuristic world, which I haven't seen done before. The character will leave the world she knows behind and be thrust into the other one with no way back. She will need to navigate this new world and rediscover the purpose she thought she lost.

March 26, 2019 - Facial Expressions Continued

Got the facial expression add-on working. It can take a video file or a FACS csv file. The csv file takes precedence if present. If a csv file is given, then it parses the file and smooths the data, then animates the expressions. If a video file is provided, it runs it through OpenFace which produces the csv file, then the addon proceeds as previously described. There are a few parameters presented to the user:

• Smoothing Window Size: the number of elements in the array to consider when smoothing out the data
• Polynomail Order: https://en.wikipedia.org/wiki/Order_of_a_polynomial
• Animation Intensity: by how much should the animation values be increased in percentage
• Vertical Gaze Intensity: the order by which to increase the vertical gaze values
• Horizontal Gaze Intensity: the order by which to increase the horizontal gaze values
• Mouth Animation: whether or not to include the mouth animation data
• Animate Face: Start the animation process
• Delete Animation: Delete existing animation

March 2, 2019 - Point of diminishing returns

Just completed some modifications to the ManuelBastioni YASP component to add some smoothing to the animation. I think it produces acceptable results for the first go. Most likely, I'll need to jump in and tweak later. But it does save a tone of time compared to if you had to go in manually and find where each phoneme is, etc. If nothing else, the mark pass of the add-on, where it marks the location of all the phonemes, provides value.

I think I got to a phase with the Automatic Lip Sync Project where any more effort I put in tweaking it would be beyond the point of diminishing returns. The only other thing worth putting effort in is making this a separate add-on and porting it to windows. I can see how this can be useful when working with makehuman characters. But I'll cross that bridge when I get to it.

The next step for me is to start using the tools I've created/modified to make a short scene. This will be key in working out the kinks from my workflow.

Here is a video I made playing around with lip-sync. The main purpose here is to show how the lip-sync and the facial animation can be combined. Note, this was recorded in real time, so it has pretty low fps.

February 1, 2019 - First Step in Automatic Lip-Syncing

well.. I finally have a program that takes a speech clip and breaks it down into its phonemes, with a start time and duration for each phoneme. The JSON file looks like this:

}, {

"word": "letter",

"start": 945,

"duration": 36,

"phonemes": [{

"phoneme": "L",

"start": 945,

"duration": 6

}, {

"phoneme": "EH",

"start": 951,

"duration": 7

}, {

"phoneme": "T",

"start": 958,

"duration": 8

}, {

"phoneme": "ER",

"start": 966,

"duration": 16

}]

}]

The next step is to take this information and convert it into animation data for a ManuelBastioniLab character. The code is here. Still needs work to make it compile and build easily. (Linux only... sorry )

January 16, 2019 - Lip-Sync update

Been looking at open source Automatic Speech Recognition (ASR) Engines. What I want to do is integrate with an ASR engine so I can run speech audio through it and generate timing information on when words (more particularly phonemes) were spoken. I can then take this information and create a first pass lip-sync animation in ManuelBastioniLab.

I looked at a couple of ASR engines one called "DeepSpeech" and the other called "pocketsphinx". The former is written in C++ and the latter is written in C. Man pocketsphinx is a whole lot easier to understand. And it has the functionality I want. Deepspeech doesn't produce timing information. Well they do, but it's not exposed in the API, which means if I need to use it, I have to take the initiative and expose this information in the API. I actually thought about it, but it's a lot more work than I'm willing to undergo at the moment.

pocketsphinx is used in rhubarb-lip-sync. However this application runs only on windows and macs (as far as I can see) and I'm not a windows user. I have seen the light and abandoned windows. In other words, I need to have something working for linux. I'm cool if I only support linux.

rhubarb-lip-sync is also a generic application designed to work with multiple different application, like after effects. Anyway, the end result for me is to develop an application which works very similarly to rhubarb-lip-sync, but is directly integrated in blender.

We'll see how that goes.

P.S. If you're interested in learning more about pocketsphinx, here is there wiki.

December 31, 2018 - Last Day of the Year Recap

2018 passed pretty quickly. It amazes me how fast time goes. The passage of time has its advantages and disadvantages. One of its advantages is it motivates people to accomplish goals. If I had all the time in the world to get something done, it's likely I won't get it done with any urgency. But I know my time is always running out. A fact which motivates me to work and get things done.

Putting philosophy aside, I think this week has been somewhat beneficial. I worked through the story with Nicole a bit more. I think we're working through the plot holes. I've also figured out something new about the ManuelBastioni Lab. It has a very extensive set of shape keys, which in theory should be a replacement for a facial rig. You know what that means, I'm using it in my proof of concept short. It's superior to Makehuman. The control it gives you over modifying the character is better, the skin shader is better, the weight painting is a lot better, the muscle system and rig are awesome. Only disadvantage is it can't be used to create small children, so I'll need to use Makehuman for that.

The next step is to fit the ManuelBastioni character with some clothes, then we're set to start building the location and moving forward with the 1 minute short. I'm excited. Hopefully, I'll have something completed in the first month of 2019.

I'm disappointed Manuel Bastioni has decided to stop supporting the lab. I understand where he's coming from. No one has showed him any support and it does take a lot of time to get this lab working properly.

I put some time into porting it to 2.80. I have taken some clothing assets from Makehuman, and I'm planning to make them available for the ManuelBastioni Lab. Maybe that'll encourage others to contribute assets, which will be all around useful.

I do think having a facial rig is useful, but I hate weight painting. I need someone to help out with that. Hopefully, we'll be able to get that project completed in 2019.

YAAM seems to be gaining some traction. I put a post on blendernation: https://www.blendernation.com/2018/12/28/free-download-asset-manager-add-on/ Got 45 shares... which ain't bad. And another dev is also working on it, so I think it could get some cool features implemented in 2019, which will make it even more useful.

I also made some small contribution to the Blender add-on code base, mainly around porting add-ons from 2.79 to 2.80. Not too shabby. Looking forward to contributing in more significant ways in the future, if I get an opportunity.

Well, signing off 2018. Happy New Year to all. And hope you accomplish your dreams in 2019.

December 27, 2018 - Open Movie Roadmap

As we draw 2018 to a close, I'm updating what I have to do on the Open Movie Project. This is not a small project by any means. It spans many fields, from software development, to writing, pre-production work, 3D production and post production. Bound to keep me busy. But baby steps.

Proof of Concept

As I mentioned before I'd like to make a 30 second to 1 minute proof of concept short. This is the first fully animated series I've ever worked on. I need to figure out a workflow that suits me. A proof of concepts should do it for me. I already have a short scene written up, different from my previous post. I'm now working on storyboarding. I figure I need to do the following:

1. Plan out the camera movement
2. Conceptualize the set and look for references
3. Build the models needed for the scene
4. Story board
5. Blocking
6. Key Poses
7. Record references
8. Splining
9. Smoothing

Obviously this is all going to be iterative. I need to really pay attention to how the final scene will work to avoid any time consuming re-work.

Software Development

In terms of Software development, there has been a few activities going on:

1. I developed YAAM
2. I submitted a few changes upstream to update Blender add-ons to blender 2.8, primarily A.N.T Landscape (currently under review).
3. I updated the Cycles Material Vault add-on to work in blender 2.8, as well as a couple of other free add-ons available on the resources page.

In 2019 I hope to accomplish the following software development goals:

1. Finish up my distributed renderer
2. Create an integrated Screenwriting add-on for blender. My vision is to work only in blender and not to have to use different software for writing screen plays, and making storyboards etc.

There will probably be other minor updates I'll have to do over 2019.

Gonna be a busy year. We'll see if we'll have any contributors on the project.

December 20, 2018 - Asset Manager Update... Holy Molly!

It's a lot of work, especially when I have to learn how to interface with the Blender Python API. It's a lot of visits to the API documentation, and looking at existing add-ons to understand how they did things. Not to mention a lot of trial and error... But I'm getting there.

I'll have it done by Christmas 2018... It'll be my Christmas gift for myself! And for whoever wants an asset manager.

I decided what to call it...

YAAM: Yet Another Asset Manager

I think it's kinda clever... yet not completely original :) It's a spin off YAML: Yet Another Markup Language.

December 14, 2018 - Test scene

Been a long week at work. Hadn't had a chance to do much on this project. But now I have. I'd like to work out the kinks from my animation workflow. To do so, I'm going to make a short scene with multiple shots. Mainly an action sequence. I figure if I'm going to do something, might as well do something difficult and hone my skills. Here is what I came up with

Story

• Story Goal: Stop a bomb from going off
• She's hanging from her leg off a building
• She wakes up and looks down to see a long drop
• The rope goes loose
• She reaches the rope just in time before it slips from her leg
• She swings with the rope until she reaches a pipe.
• When she hangs on the pipe the pipe breaks
• She hangs on a window and climbs to the roof
• On the roof, she runs through some obstacles and jumps to the next roof
• She lands and rolls then back to her feet in front of a bomb
• The time on the bomb shows less than 10 seconds left
• She opens a little door on the side of the bomb to see a few wires
• She produces a knife hesitates and cuts the blue wire.
• The time freezes at 1 second
• She breathes out in relief

December 10, 2018 - More walk cycles

I think there is an improvement, no? But going forward, I'll probably just render my test animations in EEVEE. Cycles is a lot more time consuming. But everything looks better in cycles. I think EEVEE needs different material setup . New blend file is here.

December 8, 2018 - Some more Cycles vs EEVEE comparisons

I still think cycles does a clearly better job rendering in low light. It mimics the shadows properly. I admit the materials are still not quiet there. But even without great materials cycles does a pretty good job at rendering... I'm starting to get the feeling that I might want to render the final movie in cycles. I can always have two versions. The rough copy can be in EEVEE.

December 4, 2018 - Mind Map

I was feeling a bit lost, jumping around between different tasks, which is not an issue in and of itself, but without a plan, it could lead to not accomplishing much. I decided to step back and put a task list together. I use a mind mapping tool. This task list will be continuously changing. Unfortunately, I don't have a way to actually make the mind map viewable on the site. But here is a link to download it. You can use "freeplane" for linux or "freemind" for windows to open it.

You can visit this page for a text based version of the map.

December 3, 2018 - MHX2 change

Some technical mambo jambo to follow.

If you need to import makehuman using MHX2, you'll need to follow the instructions here.

The scripts provided works fine for 2.79, but the API has changed in 2.80, so you'll get a python error when importing. I fixed the script. I wanted to push a patch into their repository, but don't know how. Until I figure it out, here is the diff. You can apply it on your version of the MHX2 script under 'scripts/addons/import_runtime_mhx2/utils.py' to get it working:

diff utils.py ../../../../2.79/scripts/addons/import_runtime_mhx2/utils.py

109c109

< ob.select_set(value)

---

> ob.select_set(action=('SELECT' if value else 'DESELECT'))

136,137c136,137

< ob1.select_set(False)

< ob.select_set(False)

---

> ob1.select_set(action='DESELECT')

> ob.select_set(action='SELECT')

November 26, 2018 - Rigging deep dive

Well using autorig or even the rigify face rig was a failed experiment. Either way, I think it'll be a good idea to refresh my rigging skills. I have done a few before, but I'm far from a pro. And I'm out of practice. Therefore, I'll take the plunge. I decided to get better acquainted with rigging. I'm not looking forward to weight painting; seems like a lot of trial and error. I updated the Resources page with more courses on Rigging. These are the best I've run into.

November 25, 2018 - Story feedback.

I had a great discussion with Nicole Panton on the story and how to improve it. Notes are here.

November 25, 2018 - Holy Crap, Rigging is hard

Well that says it all. I've been trying to do a test animation. To do that I need to get the character rigged properly and it's freakin' hard. I'm drowning in a sea of bones, facial rigs, drivers and shape keys. Just gotta keep at it.

November 19, 2018 - Expectations

I've been watching a lot of animated films lately. First to decide on the style I want and second to set my expectations. This is going to be a long haul project. Hopefully, the quality will improve over time, especially if other artists participate. But as it stands at the moment, I'll be generating my characters using Makehuman, Mixamo or ManuelBastioniLAB. Initially, I wanted to use cartoon characters similar to Disney films like Tangled or Inside Out. However, the only character generator for this is CG-Cookie's Flex Rig. I like the characters it produces, but the clothing is not great. I can go down the road of modeling clothes for the characters, but as I continued writing the story, the cartoon character look didn't seem to fit in how the story is turning out.

It appears I'm setting on a mix between cartoon and realistic looking characters. These can be produced with the character generation packages I mentioned previously, and still fit within the story. I'll probably need to rig them myself using Blender's Riggify.

My hope then is to have quality somewhere in the ballpark as video games:

• https://youtu.be/pZ15dS1ba4M
• https://youtu.be/cXflUd4d_-Y

These are still pretty high quality, so we'll see how it goes. Animation plays a big part on how real something looks. Some of the things I'll defer for later in the project are things like clothing/hair simulation. I'll start by using mesh clothing, then as I get better and faster I'll add clothing and hair simulation.

Of course all this is up for change. I'm still at the very beginning.

November 19, 2018 - Thoughts

I moved the below paragraph from my initial entry on this page. I tried to keep the first entry as an introduction without any philosophical thoughts. But below is my reasoning for doing this project and how it evolved from a short film to an episodic series.

----

The main reason I was looking to do something short was because I feared not getting the project done. I hate starting projects and not finishing them, gives me a sinking feeling in my stomach. I asked myself what's the motivation of doing this project? Do I want to be a famous filmmaker? It would be nice, but no. After thinking about it for a while the only reasonable answer was to make something I enjoy. To be honest, I want to do something I'm passionate about. Get good with Blender. And express my creative energy. If that results in a bunch of people watching it, then cool. If not, then at least I would've made something I'm proud of. People's opinions should be secondary. Doesn't mean I don't take other's input seriously, but I don't want people's opinions to shape my path.

I ended up settling on doing a long form episodic series. The season would tell an entire story, but it will be divided into a series of episodes, which would be between 7-10 minutes.

I'm currently writing the story and we'll see how it goes.

----

November 16, 2018 - Blender 2.8

I'm gonna geek out a bit here. Blender 2.8 is shaping up and I'm pretty excited about it. The coolest feature I'm looking forward to is EEVEE; the built-in real time renderer. I already tried it and it gives pretty good results... And did I mention... It's freakin' real time. What does that mean? It means that I can work on the animation and have the final result in EEVEE run in realtime. Of course cycles would probably provide higher quality results, but at the cost of compounded render time. So I'm planning to have all my effects, textures and materials in both EEVEE and Cycles.

Of course there are some disadvantages I have to be aware of. First off, Blender 2.8 is still Beta. So there are bugs. However the official release is coming out early next year, so good news there. Many of the add-ons are still not Blender 2.8 compatible. This is going to be a pain. There are a few add-ons I use, which probably won't work in blender 2.8.

My workflow will need to accommodate both Blender 2.79 and Blender 2.8. I'll have to use some of the add-ons in blender 2.79 and then import the result in 2.8. I'll also look at modifying the add-ons to work in Blender 2.8, which means I'll have to get down and dirty with Python programming. Should be interesting.