Mailbag: IF for Reinforcement Learning

Hi Emily

I’m a PhD student working with Prof. Mark Riedl at Georgia Tech and Microsoft Research Redmond. I am currently working on making AI agents (specifically using reinforcement learning) that play interactive fiction games (text-adventure games in the vein of Zork) in a non-game specific, generalizable way.

I was advised by Prof. Janet Murray that you would be the right person to help answer a question I had regarding these games, given your expertise in interactive fiction. If you have a list of such games (e.g. those given here https://github.com/microsoft/jericho#supported-games), is it possible to identify a subset of maybe ~10-15 of them that reasonably cover a majority of all interactive fiction games in terms of game structure, i.e. linearity of progression/score accumulation from the perspective of a learning agent? If it is possible, what would this set look like? Any insight at all would be great.

Nice to hear from you — I’ve been keeping an eye on this space as people have been publishing about it recently.

I’m not sure there’s a perfect answer to this, since IF is hugely varied in how it handles world model, score, pacing, etc. Also, your list here skews very much towards early interactive fiction, which means it doesn’t cover some of the formal experiments that came along later.

I also don’t remember how score works in all the games in this list — some of them I’ve not played, or played a long time ago.

However, with that in mind, here are a couple of categories that represent some fairly standard game structures:

Short or medium game in which score is given out rarely — Lost Pig (max 7)

Short or medium game in which score is given more frequently — Meteor etc. (max 30), Balances (max 51)

Long game in which score is distributed fairly frequently throughout — Adventure, Zork; possibly Enchanter and Sorcerer also; Anchorhead, as I recall

And from your list, I recall these being ones that might pose an interesting challenge:

Curses — it’s long, it’s complicated, it does have a scoring system which it doles out gradually, and it also does a trick (if I’m remembering right) where it actually at one point deducts score from the player again. 

Wishbringer — this one’s interesting because there’s a scoring system that reacts to how many times you’ve used the magic stone in the game — so the more you use wishes, the easier the game becomes, but the lower your final score.

Hunter, in Darkness — doesn’t keep score. There’s also a procedurally generated maze in this, which I would expect to make it very challenging indeed.

Thinking about games not on your list, here are some other formal extremes that might be interesting to try to reason about; all of these can be found on https://ifdb.tads.org/ and in most cases they’re available for download.

ASCII and the Argonauts — an intentionally short and simple game that gives a bunch of +1 rewards for doing basic tasks; the relatively small verb set might make it easier than some of the other games.

Aisle — a game that takes one move to play, and for which many different verbs are available; there’s also no score. It’s hard to imagine how one would use reinforcement learning on this, but it represents one extreme that might be valuable for purposes of thought experiment.

Adventurer’s Consumer Guide — as I recall this one gives out a pretty steady stream of +1 point rewards, rather than only a few or only rarer rewards, so it might be a nice counterpoint to some of the others.

Savoir-Faire — a game of mine, and I suggest it just because I happen to know it well enough to know how the rewards work; there are frequent opportunities for scoring and some rewards are bigger than others.

Bronze — a game that I wrote that keeps track of how many rooms you’ve explored and triggers certain narrative events when you’ve found more of the space, so you could use the explored-rooms count as a secondary signal to score and probably get some useful reinforcement out of that aspect as well.

Captain Verdeterre’s Plunder — gives you a score based on how much loot you managed to rescue off a sinking ship before it goes under. Genuinely an interesting optimization problem; human players have competed to try to come up with the highest-score possible traversal.

Journey to Alpha Centauri in Real Time — as the name would suggest, this takes place over a certain amount of elapsing real time and therefore it’s not possible to finish, because it’s representing a very long journey in space.

Rematch — a game in which the challenge is to figure out a single very long command that will solve the game in a single move, and in which there is a cyclical pattern to the initial world set-up. (I think this one is not a z-machine game, so it might not work with Jericho.)

Zero Sum Game — starts with a score and counts down to zero (but this may be less interesting than the others since you could just reverse the sign of the signal and wind up with something equally valid).

Hadean Lands — fiendishly hard puzzle game, in which instead of score you’re gaining access to lots of objects which could arguably be used as a proxy for progress. Also features areas where the player has to do similar things in slightly different ways.

Former Canon

There are quite a few games from the 1990s and early 2000s that are still routinely mentioned in roundups of key interactive fiction. And then there are others that felt equally or almost equally important at the time, that were commonly discussed in theoretical conversations about the state of the art, but that tend not to be remembered now nearly so often.

This list is an attempt to surface some of those games, either for interest value or because some of them really are still a lot of fun to play and worth looking back at.

John’s Fire Witch (John Baker, 1995) — from a modern perspective, this is arguably another one of any number of IF Comp-length text adventures with a pretty standard fantasy puzzle concept. At the time, there weren’t very many short games in this genre, so the concept of something accessibly-sized was notable.

Rematch (Andrew Pontious, 2000) — occasionally still mentioned by people who have been around a while, Rematch was an experiment in player expressivity and puzzle design that presented an extreme edge case of what is possible with parser IF. I tend to think of it as a bookend to Aisle: both are one-move games with a story to tell, both push the range of what we expect from parser interaction. But where Aisle simply recognizes lots and lots and lots of commands, Rematch recognizes very long and complex commands, where you’re requesting other people to do multi-part tasks for you.

Mystery Science Theatre 3000 Presents “Detective” (C. E. Forman, Graeme Cree, and Stuart Moore) — Interesting for the way it framed another work, though also in some ways a bit mean-spirited, this piece reimplements another, older work and then provides a running commentary upon that work’s failings. (It is neither as kind nor as inventive as the much more recent Re: Dragon that reframes a different entire game. At the same time, there was something interesting and daring about this: perhaps because, up to around this point, people tended to regard IF as so much effort to create that vaguely troll-y joke games weren’t that common, and neither were games that existed primarily as commentary.)

Pick Up the Phone Booth and Die (Rob Noyes, 1996) — A very short game about doing the title action and receiving the title result. But this was sufficiently entertaining to people that there were loads of spin-off joke games as well.

Sunset Over Savannah (Ivan Cockrum, 1997) — This game implemented a gorgeous beach full of meticulously interactive objects (sand to play with, underwater sequences…) and was one of the things I aspired to when I was working on simulation elements in my own games.

Worlds Apart (Suzanne Britton, 1999) — Huge and imaginative science fiction/fantasy game with many, many convenience features for the player. If you played Blue Lacuna and now want more, maybe check this out.

In the End (Joe Mason, 1996) — an early example of puzzle-less IF that took on serious subject matter instead, looking at the case of a character who is facing some serious problems and trauma.

She’s Got a Thing for a Spring (Brent VanFossen, 1997) — Prior to about 2000, this game was considered a gold standard for NPC interaction. There’s a non-player character named Bob who has loads of dialogue as well as an entire routine to how he goes about his day. The game has some other notable qualities, like an interest in landscape that sort of prefigured The Fire Tower.

Delusions (CE Forman, 1996) — one of the first hobbyist IF games I played, after Curses. It’s big, and I remember it being difficult and a bit confusing, but also ambitious and interesting. It involves a protagonist who isn’t exactly what you think, and the way your perceptions break down around you hadn’t been done much before. It was also, for the time, very technically sophisticated.

Other resources:

Post-Spirit

Usually I use the first Tuesday of the month for a book review on some book about writing or game design or narrative. This month, I’ve had enough going on that I don’t have such a review ready.

Instead, some news: for a bit over three years, I’ve been at Spirit AI, first leading the Character Engine product and then for the past seven months as Chief Product Officer.

That has been an amazing experience in many ways — educational, inspiring, and requiring a huge amount of growth. I am immensely grateful for the opportunity to do that job, and very fond of the people I worked with. I still believe in the mission of both products.

That process also, however, meant more and more distance from creative, technical, and design work, and I found that I didn’t want to sustain that path indefinitely. So I have left Spirit: yesterday was my last day. (I think I’ve fixed all the places in social media profiles that say otherwise, but if you find something else I should fix, ping me and I’ll fix it.)

In the short term, I’m doing a little contracting — including to Spirit itself, and to Spirit’s clients, so that I continue to be involved with Character Engine. Also getting a bit of a rest, because this has been almost as intense as it has been enjoyable. And maybe playing some IF Comp games.

Game Writing and Narrative Design Podcasts

I’ve covered many books on game writing here over the years, and I’ve collected and linked a lot of online resources on narrative design and on the history of IF design in particular. I also recently did a post about blogs in this space; and here is one about podcasts.

Continue reading “Game Writing and Narrative Design Podcasts”

Signs of the Sojourner (Alpha, Crowdfunding)

Signs of the Sojourner is an IF-friendly work with a beautifully expressive conversation mechanic that focuses on our resources and style of interaction rather than specifics of dialogue choice.

Play through conversations by deciding how to follow your partner’s lead.

Signs of the Sojourner is a conversational deck-building game by the small indie studio Echodog, currently crowdfunding on Indiegogo and featuring writing by the excellent Kevin Snow. A polished, substantial demo is available on itch.io.

In short: I saw Kevin Snow’s pitch about this game and thought, neat, any game with Kevin’s writing is worth a look. Then I paid a couple dollars to download the demo from itch — you can of course get it for free, but it seemed polite. Then I played the demo through from start to finish, twice, getting significantly different experiences in the two playthroughs.

When I got to the end of the second play, I was having so much fun that I really felt quite sulky about the fact that this was just a demo and that I can’t play the full game until later.

Since I really want the full game to exist in maximum glory, I backed it and then came over here to tell you about why it’s cool. And as I wrote up this post, I needed a few more screenshots and wound up replaying almost all of the alpha for a third time because I was still having fun and discovering some new things about how the mechanics worked.

Continue reading “Signs of the Sojourner (Alpha, Crowdfunding)”

Patchwork Girl (Shelley Jackson) and Spatial Hypertext

Reading Shelley Jackson’s classic 1995 hypertext, from the perspective of the present. Part of a series on literary hypertext.

This is part of an ongoing series of posts looking back at literary hypertext of the 90s and early 2000s, considering both example texts and contemporary scholarship and theory. Here we look at the concept of spatial hypertext.


The move from document-centered hypertext systems to map-based hypertext systems had some unforeseen but far-reaching implications: relationships between nodes could be expressed in more than one way. Maps showed interconnectedness explicitly, usually in the form of a directed graph. But also node proximity came into play; relationships among different nodes or documents could be indicated simply on the basis of their relative location. The use of these map-based hypertext systems to author new information spaces uncovered an interesting phenomenon. Users avoided the explicit linking mechanisms in favor of the more implicit expression of relationships through spatial proximity and visual attributes…

Spatial Hypertext: An Alternative to Navigational and Semantic Links, Frank M. Shipman, III and Catherine C. Marshall, ACM Computing Surveys 31(4), 1999.

“Spatial hypertext,” in this context, does not refer to pieces that map sections of text to a physical geography. There’s a fair amount of that in modern interactive fiction, perhaps because of the cross-pollination of ideas with parser IF. Reed Underwood even wrote an article for Killscreen about spatial hypertext that does use that phrase to mean “hypertext that represents a concrete physical space.”

Here, however, we’re using the older sense: hypertext where the connections between sections are visually represented to the reader in a kind of mindmap construction. Sometimes that mindmap might happen to resemble an object or physical location, but it might just as easily have some entirely different shape, conveying an entirely different kind of relationship between elements. Clicking on different nodes of the map navigates the reader to new contents.

On the screen, spatial hypertext can look a little like a shape poem, except that the blocks themselves contain significant amounts of text, or additional block arrangements of their own. Here is Shelley Jackson’s Patchwork Girl, published in 1995:

The cover of Patchwork Girl, its arrangement of text clusters laid out like an anchor.
Continue reading “Patchwork Girl (Shelley Jackson) and Spatial Hypertext”