MIDI Music in AS3

So, for fun, I experimented with playing midi music in AS3 today, partly to see how hard it was to do, and partly to learn more about sound programming. I may have plans for this kind of stuff for future projects, but for now, it's just an experiment. Scroll to bottom if you just want to see source code.

First, why would I want to play MIDI files in AS3? Well, MIDI files are small (think kb for long songs, not mb), and very easy to manipulate. Since MIDI files contain just information about the volume, pitch, and duration (and a bunch of metadata/instruments that I wont handle, so I'll ignore in this blog post), it's trivial to edit the song, increase or decrease the tempo, and perform pitch shifts or even changing of instruments, and this can all be done dynamically. Unfortunately, for whatever reason, Flash does not support native playback of MIDI files, nor can you access the native soundbanks (instruments) to play the MIDI files from flash. I've seen some as3 projects use sockets to play the MIDI files via an external java applet, and I've seen some as3 projects that load sound bank files as well, but the first method requires an external dependency, and the second method requires loading the soundbank files for every instrument you want (which can be tens or hundreds of mb). However, there's a third option too - dynamically generating the soundbanks, and that's what I wanted to experiment with.

The idea came from as3sfxr: http://www.superflashbros.net/as3sfxr/ (which I found on the One Game A Month website). It allows you to dynamically generate sound effects with a bunch of interesting parameters. You can see some of their example sound effects on the left. However, tweaking with the parameters, I thought I could come up with some interesting instruments. You can control the start frequency and the sustain time to change the pitch and duration of a note - everything you need to play a MIDI file. To get a nice clean tone, I recommend setting everything to the defaults, and then choosing sinewave (which produces a pretty generic sinewave), but experimenting with the settings, you can create a whole set of instruments, like electronic, retro, and even real-world sounding instruments. So, I had my dynamic, tweakable "soundbank," all I needed to do was load a MIDI file and then play the generated sounds at the right times, with the right pitches and right durations.

The Instrument class handles playing a note with specified sfxr parameters. It changes the frequency and duration parameters for each note you wish to render, and then ADDS the generated sound samples to the passed buffer (at it's current position). Adding sound samples together is an approximation of two overlapping sounds playing at the same time. To be fully realistic, you would want to add the frequencies of the two clips (using a fourier transform), not the samples - but that is not only complicated, but also very computationally expensive if we're doing it on the fly, and adding the samples sounds good enough. So, if you learn one lesson from this blog post, it's that if you have two sound clips and you want to generate a new sound clip that has both of them playing at the same time, just add the samples (if you don't know what samples are, that's another topic, but feel free to ask me and I can explain it, or just look at tutorials on WAV files).

Some minor details - sfxr takes a decent amount of time to generate the samples from the parameters, so I had to cache the results for every pitch and duration the instrument came across. That sounds crazy, but computers have a lot of memory, and the same note will occur many times, so it speeds things up many times. Also, sfxr's start frequency and sustain time parameters are very unintuitive - they are numbers from 0 to 1. The static method getLengthFromS in Instrument takes an input (in seconds) and returns a sfxr value from 0-1 (or higher if it's longer than 2.something seconds, luckily I hacked sfxr to allow sustainTimes larger than 1 - I also made some other minor changes to sfxr, mainly to access private methods). The static method getFreqFromHz in Instrument takes an input frequency in hz and returns a sfxr value (clamping from 0-1 is ok here, as frequencies below 0 and above 1 sound terrible anyways). I had to see what the code was doing to figure out the mapping function, but trust me, it works, just trust these methods :P. The static method getFreqOfMidiNote in Instrument takes a midi pitch number and converts it to a frequency in hz and then calls getFreqFromHz to return the sfxr frequency for the midi pitch. This algorithm I found online, 2^((n-69)/12)*440 - basically, every 12 midi numbers is an octave, which means multiplying or dividing by two, and note 69 is an A with a hz value of 440 that's a baseline.

The Instrument class is the most important one, but we also need a way to manage tracks of notes. The Track class has a single instrument, and then an array of note start, duration and frequencies. You can add notes to it by calling addNote (they should be added in ascending order of note start for playback with MIDISound, any order is fine with CachedSound). The renderNote function renders a single note with the track's instrument to the correct position in the byte array. The render function renders ALL notes with the track's instrument to the correct position in the byte array (this can take some time). The prepare function ensures all of the notes are cached for quick playback, and runs asynchronously with a callback.

The Main class uses as3midilib (https://code.google.com/p/as3midilib/) to load the midi song and add the notes to a Track class, and then plays it. This is relatively straightforward, but MIDI files don't give you note duration, instead they have note start and note end events. So I had to keep playing notes in a hash and then when I got the note end event update the duration. I've noticed some bugs with the MIDI loading, the division value isn't always correct (which is why it's 0.6/file.division instead of 1/division), and one file had events at the wrong times, but it mainly works with a little tweaking.

Finally, I have two playback classes. CachedSound takes a ByteArray of samples and plays it in a loop. This can play a rendered midi sound from Track.render, or whatever else you want. But, the render method can take some time, so I also added a MIDISound playback class. It allows you to add tracks you want to play, prepares them all so that all of the instruments are cached, and then will render and play the full MIDI byte array at the same time (it assumes notes are in ascending order of start time to do this). It will also loop, and after the first play it will render from a cached version just like CachedSound.

That's all for this experiment. You can hear an old MIDI song of mine (composed in middle school >_>) being played with a clear sinewave here: http://fancyfishgames.com/MidiPlayer (entire swf is only 17 kb!)

You can hear the same song played using the generateBlipSelect method of SfxrParams for the instrument here: http://fancyfishgames.com/MidiPlayer/blip.html (literally change p.resetParams(); p.waveType = 2; p.decayTime = 0.15; to p.generateBlipSelect(); to in the Instrument class do this).

And the source code plus all dependencies is here: http://fancyfishgames.com/MidiPlayer/MidiPlayer.zip (undocumented, but I explained most of it above). Feel free to experiment with MIDI files and instruments! Oh yeah, the Main.status.text lines are just there to set that one line of text, feel free to remove!

If you have questions, feel free to ask!

Pandora's Box Direction

Recently, sandbox games have gained a lot of popularity. Sandbox games allow the player to be creative, and provide many options for playing and solving problems. This freedom is what makes sandbox games so interesting, but it also usually correlates to a lack of direction in the game. In linear games, the player is constantly directed, with the game difficulty slowly increasing to challenge the player and keep them engaged until the end. With sandbox games, the player is usually thrown into a world and gets to do whatever they want. But without any direction, challenge or "end," instead of keeping the player engaged, the intrigue of the game will slowly wear off and they eventually stop playing. While obviously gameplay evolution and direction could keep the player engaged longer, adding strong direction would only ruin the freedom and spirit that makes sandbox games so successful.

A very successful sandbox game that has very little direction is Minecraft. I recently played the game, and enjoyed it a lot. In the beginning, there was a simple goal: survival. There were many ways to be creative to accomplish that goal - building structures, digging trenches, mining resources, crafting equipment, growing food, etc. Eventually, my fortress became impenetrable by the enemies in the game, and direction was lost. Survival was guaranteed, and from that point on, the game was all about exploring, experimenting in the world, and being creative. The multiplayer aspect improved this part of the game, as you could show your inventions to friends and work together on building projects. However, eventually, without challenge or direction, the game started to get dull. This is not to say Minecraft is a bad game, it entertained me for a long time before it got dull. But, I think with more direction, the game could have been even better and lasted even longer. For instance, I did create a netherword portal and explore it a little out of curiosity, but given there wasn't much there of value and it was very dangerous, I stayed out of the netherword for the most part. I feel like if there was direction, a reason to enter the netherworld, that could have added a whole new part of the game where I had to leave my comfort zone and learn to deal with the new challenges and enemies of the netherworld.

A smaller, less well known game by the same developer, is called Minicraft. Minicraft was made in 48 hours, and on it's surface, it looks a lot like a 2D version of Minecraft. However, Minicraft provides direction throughout the entire game without ruining the sandbox feel. The goal of the game is not to survive, but to defeat the Air Wizard. To defeat the Air Wizard, you'll need Gem equipment, and to get Gem equipment, you'll need to go down three levels of caves, each more dangerous than the last. The game suffers from rough edges and poor balancing due to the short development time, but there is direction to the game and the difficulty increases as you progress, without FORCING the player to do anything, keeping the freedom and spirit of sandbox games.

This form of direction I like to call Pandora's Box Direction (or player-initiated direction). Whether due to curiosity or need, the player is compelled to open Pandora's Box. Like in the myth, when opened, Pandora's Box increases the challenge and difficulty of the game, forcing the player to learn how to deal with that challenge by manipulating the sandbox. Quite possibly, in order to deal with the new threat, they feel compelled to open a new Pandora's Box, helping them deal with the first threat but releasing an even bigger threat. This creates a chain of direction, that always keeps the player on their toes and continues to give them reason to build and modify their sandbox. This chain allows the developer to balance the challenges at each box, but ultimately leaves the decision of when to open the box up to the player. This kind of direction can also create a plot, where at each "box" they learn something new, and are lead slowly but surely to some final confrontation. The direction could be completely linear, but because the player is given freedom of when to open the boxes, and freedom of how to deal with the new challenges, the game retains it's creative sandbox feel. It's like recreating that exciting first stage of Minecraft many times, each time with new enemies, challenges, and resources to keep the player engaged. For example, in Minecraft, mining diamonds could unleash a dragon from underground, who is angry you stole its treasure. Because the dragon can fly and burn wooden structures, players would have to completely rethink their defenses. And perhaps the best way to slay a dragon is to use a magic system, which requires resources unique to the netherworld, opening up a whole new pandora's box.

Sandbox games without direction can still be great games, but I personally believe that the challenges should continue evolving, so the difficulty never bottoms out. A player shouldn't quit the game because they end up finding it dull, but because they have reached some climax and ending. The Pandora's Box method is the best way to achieve this while staying within the open style that sandbox games create. I plan to experiment with this method and hope others do too!