The Artistic Computer

Ever since college, I've been fascinated with machine learning, specifically, teaching computers to create art, music, and potentially even entire games. Is it possible for a computer to do what we do - be creative? Could an advanced statistical learning model, with enough data, produce art, music, stories, or even design gameplay and write code? If you're worried about machines becoming independent of humans and taking over the world, I can assure you that we're a long way away from that. However, every now and then I come back to this idea, thinking about how cool it would be if I could create a game that generated itself from an "intelligent" computer program. Once I can teach a computer to generate at least one game-element satisfactorily, I plan to build a full game around that ability.

This blog post is about my most recent attempt to teach computers to be creative. The idea was to create a program that would draw graphics by learning the probability distribution of a color given the pixel above it, to the left of it, and the pixel in the previous mipmap level of the input data (graphics made by humans). Creating a new graphic then involves drawing from the distributions from lowest detail mipmap to the highest, left to right, and top to bottom, ensuring that we always have the inputs for the distributions while generating. I liked the idea, because it mimicked what artists do - first creating the basic shape of an object, and then adding in details. Similarly, the program progresses from low detail to high detail with each mipmap level.

The Data

The first 493 pokemon used as input data for the program.

I wanted to generate something fairly retro, and I needed a lot of consistent sprites to teach the model. So, I chose to use sprites from Pokemon as my data set as they were easy to find and there were a lot of them. The question of whether a program taught using pokemon sprites generates copyrighted material or not is tricky, but this was just for fun/research. Since the model depends on the number of colors in the image (the more colors, the harder it is to learn and the more data it needs), I decided to convert the image to just four grayscale colors. I also downsized the pokemon to 32x32 pixels, all to make the data smaller and easier to learn from.

The Results

Ideally the program would be able to learn how to create infinite reasonable looking original pokemon (imagine catching them all!). Instead, the program generated what I've termed "inkblots," some of which don't look anything at all like pokemon, some of which have interesting pokemon-ish shapes if you look for them, and some of which are actually decent (if you squint a little). The quality of the results is completely subjective, so I'll let you be the judge:

36 randomly generated results. Didn't quite have the coherence and structure I had hoped, but not outright terrible either.

You can also test the program yourself here:

For those who are curious:
Color Version: http://fancyfishgames.com/Procedural/pokemon_color.html
High Res Version: http://fancyfishgames.com/Procedural/pokemon_col_res.html

Older Experiments

Back in college, I attempted to teach a computer how to compose music by feeding midi data to a Hierarchical Hidden Markov Model. The results for that were not that great, but I still have one midi file generated by it that I thought sounded nice (from a set of anime midi files): http://fancyfishgames.com/Procedural/sample.mid, and here's a mp3 version for those who have trouble playing midi files: http://fancyfishgames.com/Procedural/sample.mp3. However, this was one result from many that were less coherent, so take it with a grain of salt.

Conclusion

I still haven't made a program that can teach computers to make games, or even acceptable pixel art yet, but that doesn't mean it won't be possible one day. For now, the results are interesting given that this was just a short project, so I figured I would share. Feel free to share any experiments you have done with machine learning and games. Until my next experiment!

On Starting a New Game Project

When starting development on my newest project, Havencall, I did a lot of things differently than for games I worked on in the past, learning from my mistakes. In this post, I explain what changes I made, and why.

1. The Simplest Possible Version

Between 2008 and 2010, I worked on Aero Empire (AE), which was a massive project, combining 4 completely different styles of gameplay and an overwhelming amount of content. Our team had many ideas of what we wanted the game to be, and we did not want to give up any one of them. Aero Empire was never completed and is now on indefinite hold.

In comparison, Havencall is a much smaller, more focused project. But that is not because I had less ideas for the project to begin with. In fact, I have notes for it going all the way back to 2009, describing an epic RPG not so much smaller than AE. The first thing I did when designing the final version of Havencall was ask myself what the simplest, most compact version of the game design would be that still had the same feel and "heart" of the original. The answer was to make it a point and click game focused entirely on the protagonist and her adventure through the three worlds, rather than a complex RPG with many characters and storylines. All of the worlds themselves were also cut to their minimal form - the first world was meant to be filled with tons of space-time bubbles, each containing its own little town. Now the first world only has two space time bubbles with towns inside them, including Aura's hometown.

You might think that making all those cuts would ruin the game and its story, but actually the opposite happened. After really thinking about what made the core of the game and cutting everything else, I ended up with something much stronger, more focused and even more exciting. Looking back, a lot of the stuff I cut was actually just detracting from what I really wanted the game to be.

2. A Group of Friends

In today's internet age, it is incredibly easy to recruit strangers to work on your projects. That is exactly what I did with Aero Empire, ending up working mostly with people I had never met or even spoken to before the project. While AE had a lot of talented people involved, the team was not very efficient or focused. We would often lose or gain people, causing loss of unfinished work and requiring time to get new people up to speed on the project. Consistency was also a mess, with almost every asset created by a different team member. And when the project started falling apart, it was only those who I had a strong connection with who stuck with me until the very end. It was also those close team members who really got each other motivated and helped everyone else stay excited.

To keep Havencall consistent and organized, I decided to make it with a team of three people I am very close with - I do all of the programming work, my wife does all of the artwork, and my sister composes all of the music. Even though our team is small, a lot of the time it seems like we are much more efficient than AE's large team - mainly because there is no lost work, no stepping on each other's toes, and we all trust each other. The bottom line is, a team that feels like a group of friends works much more efficiently and smoothly. And never underestimate the power of being able to meet and talk about a project face to face. More can get communicated in a few minutes of real conversation than you could imagine.

3. Planned From Start To Finish

There is a strong desire to just jump into a project once you have the basic idea down. All the details can just be figured out as you go along right? This is what we did with Aero Empire. The design document was rudimentary at best, and we often added or removed features without ever modifying the design document. This was very bad for two reasons: first, the design of the game was never fully understood - this made it difficult to make the game flow well and be consistent. The stages with their vastly different gameplay styles just seemed slapped together, without much thought about how they interacted with each other.

We also did not realize how big of an undertaking AE was - we knew it was big, but it turned out much bigger than we imagined. This was largely because there were gaps in the design document that we overlooked because we did not fully understand the design. The other reason to have a strong design document is to keep everyone on the same page, with the same vision. Not everyone working on AE agreed on what the details of the game were, causing confusion, inconsistency and discord. And to make matters worse, when a new team member would join, the design document was not effective at helping them understand the project.

In contrast, Havencall has been designed from start to finish, with no gaps. All of the cutscenes, puzzles, and interactions are fully detailed. This keeps the game's design focused for all team members, and allows us to create something with tightly knit art, music, and gameplay, and a unified vision.

Announcing Havencall!

EDIT: Havencall is now on IndieDB

We are very excited to officially announce Havencall, our first game using the VIDE animation toolkit! Havencall will be a short (2-4 hour) point and click adventure game which tells the story of a strange girl named Aura, and her journey between worlds to find the haven of her dreams. The original idea for this game dates back to 2009, but it wasn't until recently that everything came together: the technology, the story, and the design for what is in many ways our dream game - both to play and create.

Aura being beckoned into another world by a strange glowing figure.

The game's story and progression have been written, all of the puzzles have been designed, and we're busy working on concept art and hammering out the details. We plan to release Havencall sometime in 2013, for PC, Mac, Linux, and potentially mobile devices as well. The gameplay will revolve around interacting with items and people, solving puzzles, awakening powerful abilities, and even fighting magical devices.

My hope is for the game to be a stunning audio/visual experience that brings the strange and varied worlds of Aura's universe to life, with unusual puzzles and challenges along the way. I don't want to reveal too much of the story, so I will only say that it will be a mystical, magical and even philosophical journey.

Sneak Peek: Concept art of Aura's hometown, a small primitive village completely isolated inside a space-time bubble.

Our trajectory now is to solidify the art style, music style, and prototype the puzzles to make sure we have all of the details ironed out, and then we plan to try to fund the project on Kickstarter. It will be our first time using Kickstarter, which should be an interesting experience in itself, but hopefully a good one as well.

Stay tuned for more updates on Havencall, but they may be sparse as I don't want to spoil the story! I have also set up a website for the game here: http://havencall.com/, which we will update with more content as development progresses.