This blog post is about my most recent attempt to teach computers to be creative. The idea was to create a program that would draw graphics by learning the probability distribution of a color given the pixel above it, to the left of it, and the pixel in the previous mipmap level of the input data (graphics made by humans). Creating a new graphic then involves drawing from the distributions from lowest detail mipmap to the highest, left to right, and top to bottom, ensuring that we always have the inputs for the distributions while generating. I liked the idea, because it mimicked what artists do - first creating the basic shape of an object, and then adding in details. Similarly, the program progresses from low detail to high detail with each mipmap level.
The Data
 |
| The first 493 pokemon used as input data for the program. |
I wanted to generate something fairly retro, and I needed a lot of consistent sprites to teach the model. So, I chose to use sprites from Pokemon as my data set as they were easy to find and there were a lot of them. The question of whether a program taught using pokemon sprites generates copyrighted material or not is tricky, but this was just for fun/research. Since the model depends on the number of colors in the image (the more colors, the harder it is to learn and the more data it needs), I decided to convert the image to just four grayscale colors. I also downsized the pokemon to 32x32 pixels, all to make the data smaller and easier to learn from.
The Results
Ideally the program would be able to learn how to create infinite reasonable looking original pokemon (imagine catching them all!). Instead, the program generated what I've termed "inkblots," some of which don't look anything at all like pokemon, some of which have interesting pokemon-ish shapes if you look for them, and some of which are actually decent (if you squint a little). The quality of the results is completely subjective, so I'll let you be the judge:
 |
| 36 randomly generated results. Didn't quite have the coherence and structure I had hoped, but not outright terrible either. |
For those who are curious:
Color Version: http://fancyfishgames.com/Procedural/pokemon_color.html
High Res Version: http://fancyfishgames.com/Procedural/pokemon_col_res.html
Older Experiments
Back in college, I attempted to teach a computer how to compose music by feeding midi data to a Hierarchical Hidden Markov Model. The results for that were not that great, but I still have one midi file generated by it that I thought sounded nice (from a set of anime midi files): http://fancyfishgames.com/Procedural/sample.mid, and here's a mp3 version for those who have trouble playing midi files: http://fancyfishgames.com/Procedural/sample.mp3. However, this was one result from many that were less coherent, so take it with a grain of salt.
Conclusion
I still haven't made a program that can teach computers to make games, or even acceptable pixel art yet, but that doesn't mean it won't be possible one day. For now, the results are interesting given that this was just a short project, so I figured I would share. Feel free to share any experiments you have done with machine learning and games. Until my next experiment!
You could tweak the algorithm a bit if you don't allow single pixels floating or more than one polygon so that e.g. two separate circles are impossibles.
ReplyDeleteThis would clean it up some, and there are many other ways to clean it up further, but it would still be imperfect. Besides, it's interesting to see what it comes up with without any manually input rules on what an object or shape is. The song generated also had no manually input rules on music theory.
ReplyDeleteI dont know what algorithms you use to teach the machine (would be nice if you could throw in some keywords to google cause Im interested in that topic too), but do you think some kind of algorithm that analysis outlines and reproduces them would be a good addition? When I paint stuff, outlines ussualy add a lot to the picture.
ReplyDeleteIt's tough to say what a program that analyzed outlines would do - it would probably have better outlines than this program (which only looks at pixels), but I have a feeling the outlines would be inconsistent unless it could somehow keep track of the whole shape and the individual segments at the same time, which could be interesting.
ReplyDeleteAs for how I taught the machine to make the pokemon, it's actually quite simple - it learned probability distributions for what pixel colors could appear given the pixel above, to the left of it, and in the mipmap level above. For example, if red is above it, blue is to the left of it, red is in the mipmap level above, in the source data, 50% of the pixels in that situation were red, 25% were green, and 25% were black, and so it places a red pixel there with 50% probability, green with 25%, and black with 25% (these are not the right numbers, but it gives a good idea).
Jeah I got the same feeling about the outlinebut. The segment / shape learning could turn out that it just uses old segments and rearanges them in a new way, but combined with the probability distribution, this could actually lead somewhere worth trying =)
ReplyDeleteOnly one way to find out - try it and show me the results :) .
ReplyDeleteHaha, hardware updates will always improve software - complex smart algorithms will eventually be outdone by simple brute force ones on more advanced computers!
ReplyDelete