John Romero’s Level Design Tips

Recently I’ve been doing some research on what makes a Doom level great from a gameplay standpoint. I wanted to go back to the Vanilla days and understand how the founding fathers of Doom level design did it.

Having played Ultimate Doom recently, I was captivated by the simplicity and seamless flow of E1’s techbase levels. Romero’s Knee-Deep in the Dead levels breathe layout, something not found for the most part in Sandy Petersen’s E2 levels, though they get better (E2M6: Halls of the Damned is a classic) as the player progresses through The Ultimate Doom’s The Shores of Hell.

Below is a compilation of Romero’s level design tips, including some insights from his “Devs Play” playthrough of E1M1: Hangar.

Level Design Tips

Always change floor height when changing floor textures

Nobody likes the basic square rooms with homogeneous ceiling heights. Varying floor and ceiling heights for different rooms are more pleasing to the eye.

E1M1: The small courtyard with blue floor texturing, lowered floor and raised ceiling

Use special border textures between different wall segments and doorways

This is extremely important and distinguishes amateurish levels from high quality maps. Texturing shouldn’t be treated as applying wallpaper to floors, ceilings and walls, but rather as placing different materials on a scene (rock, stone, metal, wood, etc).

E1M2 (Nuclear Plant): A metal textured beam separates 2 walls with different wall textures. Without a transition texture the wall transition doesn’t look smooth. John Romero is pretty consistent with this motif throughout his levels

Be strict about texture alignment

All mappers should be obsessive about texture alignment, except in certain situations, for example – misaligned textures on a wall that leads the player toward a secret area.

Use contrast everywhere in a level between light and dark areas, cramped and open areas

Contrast also applies to sectors with different floor and ceiling heights.

Make sure that if a player could see outside that they should be able to somehow get there

E1M1: Overlooking the Blue Armor from the small courtyard. This outdoor area is accessible via a secret entrance

Be strict about designing several secret areas on every level

If you build a map without at least one secret area, is it even a level?

Make the level flow so the player will revisit areas several times so they will better understand the 3D space of the level

Create easily recognizable landmarks in several places for easier navigation

Golden rule of level design – the “Horseshoe” layout

Romero mentions adopting a non-linear level design paradigm. The rule is simple, according to the Gamasutra articlemake sure the player is likely to look at every wall of a room. This wouldn’t have been possible had the level presented the player with a linear path. This pattern is common in every first level of Doom, Hexen and Quake as well.

Horseshoe pattern in the beginning of E1M1: Hangar. The player is forced to explore 75% of the room

Valve Software also produced some similar thought-leadership in this area documented in their developer portal as the Loop layout element. In this layout, the “Loop” guides the player back to a previously explored area of the level, and while maintaining strict linearity, provides the illusion of non-linearity and progression.

A symbolic diagram of a loop - it begins and ends in the same room.
The Loop element

Build your first level last

Romero started E1M1 early in the development of Doom, but was the last level that he completed. In this way he was able to use his learnings from the design of all other Doom levels and apply them to the first. In this progression, the first couple of levels are the weaker ones (sans the first), middle ones get better, and culminating with the stronger levels.

Don’t constrain player movement (aka don’t let detailing get in the way of gameplay)

Tom Hall wanted to detail E1M1 with movable chairs, but the chairs ended up interfering with smooth player movement. When mapping, consider the trade-off between focusing on detailing versus pure gameplay mechanics. The former can be sacrificed for the sake of gameplay.

Use fullbright textures

Use fullbright textures when appropriate. Fullbright textures ignore the surrounding light in a scene and render at full brightness to create the appearance of emitting their own light.

The light source on the left is rendered with a fullbright texture and provides the illusion of self-illumination. In a dark room, light sources can orient the player

If you want to see out Romero’s playthrough of E1M1, where he offers his insightful level design tips, check out the video below:

Cacowards 2018 Announced!

Cacowards 2018

Doomworld’s 2018 Cacowards recently came out and with it a series of amazing articles from Doomworld. You can check them out below

  1. 2018 Cacowards: The quintessential Cacowards, celebrating 25 years of Doom. Also included is an amazing write-up on Erik Alm’s legacy as a mapper and how his enduring vision influenced other legendary projects.
  2. Top 100 Most Memorable Maps: Curated list of the top 100 Doom maps produced by the Doom community. Either as single-map contributions or cherry-picked maps from multi-level WADs.
  3. Top 25 Missed Cacowards: Projects that didn’t win a Cacoward but who looking back, should have been awarded one.

Auferstehung 2 (Doom WAD) Released!

It’s been 12 years since Helldorado Team (formerly Brain Dead Studios, formerly Elioncho’s Engine team, formerly the Condor Team) announced the development of a Doom WAD – the never released Confessions from the Flesh. But today, after more than 10 years of projects sinking down the drain, prototypes never seeing the light of day, and other tech demos lost to numerous formatted hard drives, we can finally announce the release of our first 11-level Doom WAD (wow, it only took a decade): Auferstehung 2.

Auferstehung 2 is the unofficial sequel and homage to Nazi Auferstehung, named the Worst WAD of 2006 by Doomworld.

And now, from the depths of the Doom community a similar turd emerges, albeit not as worse, but still pretty bad. Following the footsteps of the predecessor’s creator (who used Slige), Helldorado Team leveraged Oblige to produce the 11 procedurally generated levels of Auferstehung 2.

Download Auferstehung 2!

Levels:

MAP01: Enter the Dominion
MAP02: The Prison of Despair
MAP03: Horrendous Temple
MAP04: Blood Shrine of the Beast
MAP05: The Nukage Shaft
MAP06: Thermal Terminal
MAP07: Pain from the Base
MAP08: Lockdown
MAP09: The Crossroads of Menace
MAP10: Arson Anthem
MAP11: Palace of Danger

Special thanks to:

  • Andrew Apted for creating Oblige and making this project possible
  • Dark Exodus for Nazi Auferstehung
  • id Software for Doom 2
  • Elioncho for emotional support

Tools used:

How to play:

  1. Drag and drop auferstehung2.wad into the .exe of your favorite Doom Source Port (for example GZDoom, Zandronum, etc). Boom-compatible port recommended.
  2. Make sure Doom2.wad is in the same folder as the Source Port’s .exe file

ARThings v1.2 Released

We made some tweaks over the past couple of weeks and finally ARThings v1.2 is out. Click on the cool Doomguy below to check it out:

Here’s a laundry list of the updates we made:

  1. Updated HUD graphics – Updated fire button graphics and added a new “switch weapon” button
  2. Removed the ability to spawn monsters by tapping on planes. This was an experimental feature pre-v1.2 that has now been removed
  3. Removed the tracking of vertical planes such as walls. This was not needed as monsters spawn on top of upward facing horizontal planes (such as floors).
  4. You can now pick up and use the Chaingun
  5. Updated spawner to spawn Shotgun and Chaingun ammo when you’re low on ammo
  6. Updated spawner to spawn 3 different types of monsters: 1) Zombiemen, 2) Chaingunners and 3) Demons
  7. Dead Zombiemen drop clips when dead
  8. Dead Chaingunners drop Chainguns when dead
  9. Added other effects such as blood and teleport fog when spawning monsters
  10. Other bug fixes

Also, here’s a video of some “in-game” action from v1.2:

Doom AI Game Logic Explained, kinda

A couple of weeks ago we released ARThings, please check it out when you get a chance. Like we mentioned previously this project began as an academic exercise to build something more interactive with ARCore, but to bring this Doom experience to ARCore we had to invest time to learn the inner workings of Doom’s AI/Game Logic system. Below is some light commentary on this experience, including some insights and lessons learned.

Picking a Doom Source Port

To port the Doom AI game logic we had to pick a Doom source port. We narrowed down the list to 2 contenders:

There’s pros and cons with each of these but ultimately we settled with Linux Doom. By doing so we could deal with less complexity. The ZDoom codebase is much more modernized but quite extensive – it not only runs Doom but also Heretic, Hexen, Strife and other “Doom clones”. Learning ZDoom meant going through a much more extensive and technically sophisticated codebase. ZDoom’s AI and gameplay code architecture was revamped to include ACS, a flexible scripting system that developers could use to extend gameplay and AI behavior without having to write additional code to be compiled with the master codebase.

Having settled with Linux Doom, we unzipped the codebase and immediately faced with an even bigger challenge – working with legacy code. Doom was written in the early 90s, and as such the code was meant to push the limits of the hardware of that era (i386, i486, etc). If you get some time, I recommend reading Fabien Sanglard’s Game Engine Black Book, he does an amazing job describing the technical limitations id Software’s faced when developing Wolfenstein 3D. These limitations drove the technical direction of both the Wolfenstein 3D and Doom codebase.

So we went deep into the Doom source code. After studying the source code for a couple of days we encountered these technical pain points:

  1. Reading C – Doom’s code is not Object Oriented. Functions are grouped into source files based on functionality. The code can be hard to come to terms with if you’ve been writing OO code for a while.
  2. Absence of Floating Point arithmetic – To cope with CPU’s lack of floating point arithmetic during the mid-90s Doom uses Fixed point arithmetic. The gist of this is that floating points are represented using 32-bit integers. 16 bits for the integer portion and the remaining 16 for the fractional part.
  3. Angles are represented using Binary Angle Measurement (BAM) – This is new territory if you’re used to storing angles as doubles (or floats) when working with computer graphics or trigonometric calculations. in BAM, angles are represented using the full range of values of a 32-bit unsigned int. The challenge with this is that some languages do not support unsigned int as a primitive data type (Java for instance). You have to provide your own wrapper functions when performing computations that result on unsigned integers. Another challenge with BAM is that to perform trigonometric calculations you have to provide your own special purpose sin/cos/tan lookup tables.
  4. Usage of C function pointers – These are used heavily in the code, sometimes to mimic Polymorphism. C function pointer code can be hard to port with the right architectural approach, especially if the language you are porting to implements similar functionality using a different programming paradigm. Again, using the Java example you can accomplish this using Lambda expressions and Method References.

The Basics – Understanding Mobjs and States

Let’s now get to the basics. To understand how the Doom AI works we need to understand how objects, or Things (mobjs), are represented in the codebase. Things are monsters, items, pickups, fireballs, projectiles, or anything that is represented by a sprite in the game. For example, when you shoot a monster and you see a blood effect rendered as a sprite, the blood effect is a mobj.

Things are represented by the mobj_t struct defined in p_mobj.h. mobjs and have attributes such as (x,y,z) position, (x,y,z) momentum (used to simulate physics) and many other object properties.

Side note: The Doom coordinate system treats Z as the vertical “up” axis, as opposed to the Y axis. If you are coming from the OpenGL world this might be hard to stomach.

For the purposes of AI, a mobj (I’ll be using mobj and Thing interchangeably) functions as a finite state machine. Through its lifespan a mobj will cycle through various states as determined by the core engine AI logic. A mobj’s current state is kept as a reference to a State object:

state_t* state;

The declaration of state_t is defined in info.h and all state definitions in the game are stored as a global array:

typedef struct
{
  spritenum_t	sprite;
  long		frame;
  long		tics;
  // void	(*action) ();
  actionf_t	action;
  statenum_t	nextstate;
  long	        misc1, misc2;
} state_t;

extern state_t	states[NUMSTATES];

The state object has the following attributes and characteristics:

  • sprite – Sprite to use to render the mobj for this state
  • frame – Animation frame to use for this state
  • tics – How many tics to run this state for (We will explain the concept of tics in the next section)
  • action – Pointer to the function to run when transitioning to this state
  • nextstate – State to transition to when this state is over
  • misc1 – Miscellaneous attribute
  • misc2 – Miscellaneous attribute

The list of all state numbers for all the different types of mobjs in the game are defined in the statenum_t enum in info.h.

To decide what states correspond to each mobj type there’s a structure called mobjinfo_t in info.h that contains this information. The engine stores this as a global array as well:

typedef struct
{
    int	doomednum;
    int	spawnstate;
    int	spawnhealth;
    int	seestate;
    int	seesound;
    int	reactiontime;
    int	attacksound;
    int	painstate;
    int	painchance;
    int	painsound;
    int	meleestate;
    int	missilestate;
    int	deathstate;
    int	xdeathstate;
    int	deathsound;
    int	speed;
    int	radius;
    int	height;
    int	mass;
    int	damage;
    int	activesound;
    int	flags;
    int	raisestate;
} mobjinfo_t;

extern mobjinfo_t mobjinfo[NUMMOBJTYPES];

Let’s provide an example with one of the Doom monsters, the Zombieman.

Here’s how the Zombieman’s mobjinfo is defined in the info.c file. Note that the Zombieman’s mobj type is MT_POSSESSED:

mobjinfo_t mobjinfo[NUMMOBJTYPES] = {
   .
   .
   .
   {
	// MT_POSSESSED
	3004,		// doomednum
	S_POSS_STND,	// spawnstate
	20,		// spawnhealth
	S_POSS_RUN1,	// seestate
	sfx_posit1,	// seesound
	8,		// reactiontime
	sfx_pistol,	// attacksound
	S_POSS_PAIN,	// painstate
	200,		// painchance
	sfx_popain,	// painsound
	0,		// meleestate
	S_POSS_ATK1,	// missilestate
	S_POSS_DIE1,	// deathstate
	S_POSS_XDIE1,	// xdeathstate
	sfx_podth1,	// deathsound
	8,		// speed
	20*FRACUNIT,	// radius
	56*FRACUNIT,	// height
	100,		// mass
	0,		// damage
	sfx_posact,	// activesound
	MF_SOLID|MF_SHOOTABLE|MF_COUNTKILL, // flags
	S_POSS_RAISE1	// raisestate
   },
   .
   .
   .
};

The attributes in the mobjinfo object define the states to transition to and the sound effects to play for a particular mobj type when entering the following core mobj states handled by the engine:

  • Spawn State – state to transition to when the mobj is spawned in the game. This is the “idle” state where the monster just stands and does nothing, checking its field of view until it spots a player to chase and attack.
  • See State – state to transition to when the mobj spots an enemy (in this case the player) and should chase it. Inside the code this is sometimes referred to as the “Chase” state.
  • Pain State – state to transition to when the mobj takes damage
  • Melee State – state to transition to when the mobj is ready to perform a close range melee attack. Only applies to mobjs that perform melee attacks. For the Zombieman, notice that the meleestate attribute is “0”, which corresponds to the NULL state, since the Zombieman doesn’t have a melee attack.
  • Missile State – state to transition to when the mobj is ready to perform a ranged attack. Only applies to mobjs that perform ranged attacks such as a Zombieman firing his rifle or a Cacodemon belching a fireball.
  • Death State – state to transition to when the mobj is killed
  • XDeath State – state to transition to when the mobj is killed via explosion (gibbed)

Now let’s look at the complete list of Zombieman state definitions as defined in info.c:

state_t	states[NUMSTATES] = {   
    .
    .
    . 
    {SPR_POSS,0,10,{A_Look},S_POSS_STND2,0,0},	// S_POSS_STND
    {SPR_POSS,1,10,{A_Look},S_POSS_STND,0,0},	// S_POSS_STND2
    {SPR_POSS,0,4,{A_Chase},S_POSS_RUN2,0,0},	// S_POSS_RUN1
    {SPR_POSS,0,4,{A_Chase},S_POSS_RUN3,0,0},	// S_POSS_RUN2
    {SPR_POSS,1,4,{A_Chase},S_POSS_RUN4,0,0},	// S_POSS_RUN3
    {SPR_POSS,1,4,{A_Chase},S_POSS_RUN5,0,0},	// S_POSS_RUN4
    {SPR_POSS,2,4,{A_Chase},S_POSS_RUN6,0,0},	// S_POSS_RUN5
    {SPR_POSS,2,4,{A_Chase},S_POSS_RUN7,0,0},	// S_POSS_RUN6
    {SPR_POSS,3,4,{A_Chase},S_POSS_RUN8,0,0},	// S_POSS_RUN7
    {SPR_POSS,3,4,{A_Chase},S_POSS_RUN1,0,0},	// S_POSS_RUN8
    {SPR_POSS,4,10,{A_FaceTarget},S_POSS_ATK2,0,0},	// S_POSS_ATK1
    {SPR_POSS,5,8,{A_PosAttack},S_POSS_ATK3,0,0},	// S_POSS_ATK2
    {SPR_POSS,4,8,{NULL},S_POSS_RUN1,0,0},	// S_POSS_ATK3
    {SPR_POSS,6,3,{NULL},S_POSS_PAIN2,0,0},	// S_POSS_PAIN
    {SPR_POSS,6,3,{A_Pain},S_POSS_RUN1,0,0},	// S_POSS_PAIN2
    {SPR_POSS,7,5,{NULL},S_POSS_DIE2,0,0},	// S_POSS_DIE1
    {SPR_POSS,8,5,{A_Scream},S_POSS_DIE3,0,0},	// S_POSS_DIE2
    {SPR_POSS,9,5,{A_Fall},S_POSS_DIE4,0,0},	// S_POSS_DIE3
    {SPR_POSS,10,5,{NULL},S_POSS_DIE5,0,0},	// S_POSS_DIE4
    {SPR_POSS,11,-1,{NULL},S_NULL,0,0},	// S_POSS_DIE5
    {SPR_POSS,12,5,{NULL},S_POSS_XDIE2,0,0},	// S_POSS_XDIE1
    {SPR_POSS,13,5,{A_XScream},S_POSS_XDIE3,0,0},	// S_POSS_XDIE2
    {SPR_POSS,14,5,{A_Fall},S_POSS_XDIE4,0,0},	// S_POSS_XDIE3
    {SPR_POSS,15,5,{NULL},S_POSS_XDIE5,0,0},	// S_POSS_XDIE4
    {SPR_POSS,16,5,{NULL},S_POSS_XDIE6,0,0},	// S_POSS_XDIE5
    {SPR_POSS,17,5,{NULL},S_POSS_XDIE7,0,0},	// S_POSS_XDIE6
    {SPR_POSS,18,5,{NULL},S_POSS_XDIE8,0,0},	// S_POSS_XDIE7
    {SPR_POSS,19,5,{NULL},S_POSS_XDIE9,0,0},	// S_POSS_XDIE8
    {SPR_POSS,20,-1,{NULL},S_NULL,0,0},	// S_POSS_XDIE9
    {SPR_POSS,10,5,{NULL},S_POSS_RAISE2,0,0},	// S_POSS_RAISE1
    {SPR_POSS,9,5,{NULL},S_POSS_RAISE3,0,0},	// S_POSS_RAISE2
    {SPR_POSS,8,5,{NULL},S_POSS_RAISE4,0,0},	// S_POSS_RAISE3
    {SPR_POSS,7,5,{NULL},S_POSS_RUN1,0,0},	// S_POSS_RAISE4
    .
    .
    .
};

Before diving deeper into these states, let’s go back and cover the tics concept that I mentioned briefly.

Gametics and Thinkers

Doom doesn’t manage the concept of game updates in terms of milliseconds since the last game update. A gametic is the equivalent of a single game update – basically one iteration of the Doom game loop. During this iteration the engine performs one frame update and executes basic game loop activities such as capturing input, executing gameplay logic and rendering to the framebuffer.

Doom’s engine is clamped at 35 tics per second (35 iterations of the Doom game loop every second), so a single tic should take no longer than 1000/35 = 28.57 milliseconds. This is the behavior of Vanilla Doom, however some source ports have removed this technical restriction and allow uncapped framerates.

To understand the game loop, lets look at some code. The Doom main game loop makes a call to the G_Ticker() function in g_game.c. A call to P_Ticker() (defined in p_tick.c) is made as long as the game is running a level (gamestate=GS_LEVEL):

//
// G_Ticker
// Make ticcmd_ts for the players.
//
void G_Ticker (void) 
{    
    .
    .
    .
    // do main actions
    switch (gamestate) 
    { 
      case GS_LEVEL: 
	P_Ticker (); 
	ST_Ticker (); 
	AM_Ticker (); 
	HU_Ticker ();            
	break; 
	 
      case GS_INTERMISSION: 
	WI_Ticker (); 
	break; 
			 
      case GS_FINALE: 
	F_Ticker (); 
	break; 
 
      case GS_DEMOSCREEN: 
	D_PageTicker (); 
	break; 
    }
}

P_Ticker() handles game updates for the following:

  • All Players in the game (via a call to P_PlayerThink)
  • Mobjs (via call to P_RunThinkers)
  • Other things that we don’t care about right now
//
// P_Ticker
//

void P_Ticker (void)
{
  .
  .
  .
  for (i=0 ; i<MAXPLAYERS ; i++)
	  if (playeringame[i])
	      P_PlayerThink (&players[i]);
			
  P_RunThinkers ();
  P_UpdateSpecials ();
  P_RespawnSpecials ();
  .
  .
  .
}

Let’s digest all of this for a moment. You might think that to run game updates for all mobjs in the game you would just call an update() method passing the mobj as a parameter and do this for every single mobj in the game.

Doom handles game updates for mobjs a bit differently. Every mobj in the game has a Thinker object that manages the execution of game updates per frame for that specific mobj. The structure of a Thinker object looks like this (declared in d_think.h):

// Doubly linked list of actors.
typedef struct thinker_s
{
    struct thinker_s*	prev;
    struct thinker_s*	next;
    think_t		function;
    
} thinker_t;

As mobjs spawn in a level, their Thinkers are created and added to a doubly linked list (hence the prev and next pointers). This happens in the P_SpawnMobj() function (in p_mobj.c):

//
// P_SpawnMobj
//
mobj_t*
P_SpawnMobj
( fixed_t	x,
  fixed_t	y,
  fixed_t	z,
  mobjtype_t	type )
{
    .
    .
    .

    mobj->thinker.function.acp1 = (actionf_p1)P_MobjThinker;	
    P_AddThinker (&mobj->thinker);

    return mobj;
}

Likewise, as mobjs are removed from a level (for example a projectile mobj hits a monster and needs to be removed after the explosion), their Thinkers are removed from the doubly linked list (P_RemoveMobj function in p_mobj.c):

//
// P_RemoveMobj
//
mapthing_t	itemrespawnque[ITEMQUESIZE];
int		itemrespawntime[ITEMQUESIZE];
int		iquehead;
int		iquetail;

void P_RemoveMobj (mobj_t* mobj)
{
    .
    .
    .
    
    // free block
    P_RemoveThinker ((thinker_t*)mobj);
}

Every Thinker object has a reference to a function (attribute think_t function) that executes an mobj’s game updates. This function is P_MobjThinker() in p_mobj.c:

//
// P_MobjThinker
//
void P_MobjThinker (mobj_t* mobj)
{
   // We'll go through this code later
}

P_RunThinkers() simply iterates through each mobj’s Thinker and executes its Thinker function – P_MobjThinker():

//
// P_RunThinkers
//
void P_RunThinkers (void)
{
    thinker_t*	currentthinker;

    currentthinker = thinkercap.next;
    while (currentthinker != &thinkercap)
    {
	if ( currentthinker->function.acv == (actionf_v)(-1) )
	{
	    // time to remove it
	    currentthinker->next->prev = currentthinker->prev;
	    currentthinker->prev->next = currentthinker->next;
	    Z_Free (currentthinker);
	}
	else
	{
	    if (currentthinker->function.acp1)
		currentthinker->function.acp1 (currentthinker);
	}
	currentthinker = currentthinker->next;
    }
}

We can think of P_MobjThinker() as the function called for each game state update for each mobj. This function will take care of movement, physics simulation and cycle between AI states.

We can summarize the above behavior with the following pseudocode:

G_Ticker() // g_game.c
{
	switch(gamestate)
	{
		case GS_LEVEL:
			P_Ticker() // p_tick.c
			{
				P_RunThinkers() // p_tick.c
				{
					// Iterate through all thinkers in level (linked list)
					foreach(thinker in thinker linked list)
					{
						function = thinker.function;
						// Execute thinker function
						// Runs P_MobjThinker() function in p_mobj.c
						function.run();
					}
				}
			}
	}
}

In the next article we’ll continue our deep dive into Doom AI game logic and the technical details of P_MobjThinker().

While you wait for the next article we recommend reading:

ARThings Released – Augmented Reality Doom

We recently released a Doom experience for Android’s ARCore titled ARThings. It was originally titled ARDoom but after receiving hate mail from the Google Play admins informing me that I couldn’t use the brand name “Doom” as part of the app title, I changed the name.

Click below to check it out!

We’re currently on v1.1 which has the following features:

  • Click on a horizontal tracked plane to spawn either 4 shotgun shells, a Zombieman or a Heavy weapon dude
  • Shotgun shells, Zombiemen or Heavy weapon dudes spawn randomly on detected horizontal planes
  • Blast away Zombiemen and Heavy weapon dudes with the proverbial Doom shotgun

Some technical facts:

Special thanks to:

  • John Carmack for answering some questions via email related to framerate clamping in the Doom Linux codebase
  • Fabien Sanglard for letting me proofread his upcoming Doom Game Engine Black Book. This was instrumental in helping me understand Doom monster AI

Herbert’s Whipping Star: A Conceptual Work

Frank Herbert’s Whipping Star is the first installment of the ConSentiency novels and I took it up a couple of weeks ago mainly because I wanted to read the pre-quel to Herbert’s The Dosadi Experiment. It’s a quick read – the mass market paperback amounts to a mere ~180 pages so there is not much to talk about. However, in providing my thoughts on this book I am going to make some overarching assumptions – I’m going to blatantly present the hypothesis that the majority of Whipping Star readers consumed a subset of the Dune novels prior to picking this one up. It doesn’t seem likely that this would be a reader’s first Herbert novel – mainly because the majority of his non-Dune books didn’t get a lot of exposure and were not award winners (Dune won the Hugo for Best Novel in 1966 and Children of Dune was nominated for the same in 1977). Thanks to the tremendous success of Dune in the mid-60s, all of Herbert’s pre and post Dune works were able to piggyback on Dune’s success. You can see this very clearly on the paperback covers of all the non-Dune books – they clearly mention Frank Herbert as “Author of Dune and <insert-another-dune-chronicle-book-name-here>”.

That being said a big chunk of non-Dune book readers could be placed in the “Herbert Fan” bucket – readers who hungered for more ideas and universes beyond Dune and who hoped that Herbert could replicate the success he achieved with Dune somewhere else. I think Herbert did pretty good with the WorShip and ConSentiency universes in general, however, I expected a little bit more from Whipping Star as the first installment of the ConSentiency series – this small work was supposed to set the stage for The Dosadi Experiment, but it lacked in certain key areas that could have been improved to achieve some first-class world building.

Whipping Star is a short conceptual work with the following background – in the future humanity has been able to co-exist peacefully with other sentient species such as the Gowachin, LaclacWreavesPan SpechiTaprisiots, and Caleban. To bring order to this hot pot of civilizations, a federated government called the ConSentiency is established. However, this new government brought its own challenges – they were so efficient at government that laws were passed faster than you could change your underwear so another organization had to be created to slow down the pace of government and add red-tape – the Bureau of Sabotage (BuSab). In addition, humanity has been able to travel to any point in the universe thanks to the gifts of the Calebans, who are able to open “Jumpdoors” to travel from point A to point B anywhere in the universe.

The novel’s story deals with the attempt to save a female Caleban called “Fannie Mae” from the hands of Mliss Abnethe, a sadist woman with a penchant for plastic surgery who has a binding contract with Fannie Mae. The contract stipulates that Fannie Mae can be physically hurt by Abnethe and that they get some sort of mutual benefit from it (however I’m not sure what benefit Fannie Mae gained from it). The narrative centers on Jorj X. McKie’s (a BuSab agent) efforts to track and stop Abnethe, who is using jumpdoors at random intervals to get next to Fannie Mae and hurt her. A big portion of the book centers on dialogues between Fannie Mae and McKie, who interrogates the Caleban to find more information on Abnethe’s intentions and whereabouts.

Cover of the Berkeley edition, July 1981 (Fourteenth printing). Depicted here is the “Beachball”, which serves as the home of the Caleban Fannie Mae. Believe it or not this is where the majority of the action takes place in the book.

Whipping Star reads like a classic non-Dune novel in the sense that it doesn’t have any of the philosophical heaviness or atmospheric bleakness that Herbert is known for. In fact, it even has some comical dialogues that you will not find in any of his Dune novels (for example all of the conversations with Tuluk). The pace is quick and it centers mainly on dialogues, events and actions performed by the characters, not on character’s thoughts, ideas or moral dilemmas. The conversations between McKie and Fannie Mae are particularly interesting – these exchanges are thought-provoking because McKie finds communication with the Caleban excruciatingly frustrating due to world-view differences between humans and Calebans. With these dialogues Herbert offers some examples of the inherent difficulties faced when different sentient species try to communicate – this is the central theme of the Stanislaw Lem’s Solaris, however Whipping Star seems to emphasize more of the semantical difficulties and less the philosophical ones.

In terms of world-building, I expected the text to explore a little bit more the different cultural and societal aspects of the co-existence between all the sophonts of the ConSentiency. However I only found bits and pieces of it due to the short length of the book (a total of 188 pages). Had the book been longer, Herbert would have had better luck setting the stage for this universe. Just to keep in mind – Herbert is not here to hand-hold you. He is very infamous for this habit. That is why the first ~100 pages of Dune make for very difficult reading. Beginning with page 1 of Whipping Star Herbert throws at the reader terms, names and concepts that are relevant to the ConSentiency universe but which have no context for the reader. This is what makes Herbert’s books both frustrating and enjoyable – you need to put in the leg work in the very beginning to gather all the bits and pieces. Once you break through this barrier you basically unlock the wonders of the book’s universe – this makes for a very rewarding reading experience.

I recommend this book if you are looking to venture into The Dosadi Experiment of if you want to explore a Herbert novel with a somewhat lighter touch. However, if you want to read something with Herbert’s usual brutal tone and mood, stay away.

Destination: Void – A Trip From Hell

I came across this book by accident. I had been browsing for computer games to play and ended up in Sid Meier’s Alpha Centauri Wikipedia page. There I read that Brian Reynolds (Alpha Centauri lead designer) had gained inspiration by reading Frank Herbert’s The Jesus Incident and Hellstrom’s Hive. I later found out that the former was the second book of Herbert’s WorShip series – Destination: Void being the official “prequel” (Book 1) to the books in the Pandora Sequence (made up of The Jesus Incident and all the others that came after). I decided to give Destination: Void a read in order to get into this universe.

Destination: Void is not the typical Herbert book by any means. Fans of Dune will not feel at home, mainly because Destination: Void is the polar opposite of what Dune tried to be. In Dune Herbert purposely suppressed technology in order to focus on the future of humanity, not the future of humanity’s technology. Because of this Dune has been able to endure. However, with Destination: Void, Herbert ventured into the the realm of hard sci-fi and in doing so made some daring steps into domains that are not his forte – cybernetics, hardware and AI.

In a nutshell the book deals with the trials and tribulations of a crew of 4 as they manually operate a ship en-route to Tau Ceti carrying thousands of hibernating colonists. In the opening scene the crew is faced with a problem – each of their 3 on-board AIs, called Organic Mental Cores (OMCs) either committed suicide or had to be disconnected due to rampancy. Due to this the crew is forced to manage ship functions and operations manually – a repetitive and stressful task impossible to perform throughout the while to Tau Ceti. Following this the crew receive clear orders from project management based on the moon: “Build a conscious AI that can automatically manage the ship and go into hibernation for the 400-year trip to Tau Ceti”.

Following this order from the project director, the narrative breaks into a series of philosophical, moral and religious discussions between the 4 crew members surrounding the main question of “what is consciousness?”. The novel is actually a 200+ page-long Socratic dialogue (and by Socratic I mean that in each of the conversations between the crew members there’s always someone trying to be a smart-ass) on the definition of consciousness masqueraded as a sci-fi paperback. The narrative follows this basic framework:

  1. Crew members have philosophical/moral/technical/whatever debate on the definition or origin of consciousness
  2. Crew members disagree with each other
  3. Crew members develop paranoia and become distrustful of each other or distrustful of project management motives
  4. Something malfunctions in the ship and the crew has to repair it
  5. One of the crew members explains the technology behind building a self-aware AI and builds a piece of aforementioned AI using 1960s computer hardware – note: some of the components are actual 1960s hardware and others are blatanly made up by Herbert (for example ENG MULTIPLIERS). These sections are basically the most brutal and unreadable portions of the book. Herbert here is purposely trying to confuse the common paperback reader who has absolutely no idea about hardware or cybernetics. Actually, something very upsetting about the book is that by Herbert coming up with his own pseudo-technology (yes, ENG MULTIPLIERS), he is basically making himself impervious to criticism and scientific scrutiny. Since only he knows what the hell an ENG MULTIPLER is, nobody can tell him what it’s not or how it should function. Had Herbert tried applying actual 1960s hardware technology he would have been scolded for his lack of scientific rigor
  6. Crew members receive a message from project management and react to it
  7. Go to #1

While some might find the philosophical discussions intriguing, these discussions mainly detract from the tension that this novel could have had, and the actual moments of tension are restricted to situations when crew members develop various paranoias or question the motives of project management when they receive orders from the moon. The other moments where tension can be found are when things break in the ship that need to be fixed – mainly because you never know if these are actual malfunctions or if they were pre-programmed by project management as some sort of morbid prank or if they were caused by the AI-in-the-making (called “The Ox” in the book), which is running in “TEST MODE”. All in all these moments of tension amount to perhaps 10% of the book.

The cover of the revised December 1978 Berkley Edition – “supposedly” revised by Frank Herbert and updated with more up to date technology concepts based on new and improved 1970s technology! This cover is disturbing in what it portrays – Bickel (the engineer) sleeping soundly, dreaming peacefully about the new set of ENG MULTIPLIERS he’s going to add to the AI the next day.

If you ignore the numerous philosophical debates and skim over Herbert’s techno-jargon there is some room left to enjoy the moments of tension. The techno-babble can be easily ignored since it doesn’t add any value to the story whatsoever – in fact the quasi-science depicted here is less offensive than most Golden Age sci-fi descriptions of “coal-powered space ships”. Since so little background information is revealed about the project’s mission, the reader is left to put together the missing pieces and figure out what the project goals really are. Parts of the novel remind me vividly of J. G. Ballard’s short story Thirteen to Centaurus. In this story crew members are made to believe they are on a journey to Alpha Centauri when in reality they are on Earth playing the role of guinea pigs in a experiment to test the psychological effects of deep space travel.

I won’t recommend this book to the casual sci-fi fan looking for an easy read. If you can make it half-way through the book you will be rewarded with some fine moments of tension and anticipation – these are mostly confined to the second half of the book. Also, I believe the book’s ending was entertaining and gave me some moments of contemplation. In terms of the science, like I mentioned previously, most of it can be ignored but if you are a purist and need to read everything line by line you will find some gems scattered here and there that cover AI topics that are very relevant today. For example, Herbert mentions the idea of creating an AI from scratch and producing the equivalent of a dumb infant with 0 experience and 0 instincts. He then mentions the challenges faced with “training” an infant and providing a superset of past experience data points to serve as the AI’s experience. If you are familiar with the training of Machine Learning models, it’s amazing that rudimentary machine learning concepts are present in Herbert’s 1960s writings – considering his lack of formal technical or computer science education.

I will most likely continue exploring with the WorShip universe, but mostly because I read that the storyline of Bungie’s Marathon Trilogy was heavily influenced by the rampancy concepts presented in The Jesus Incident.

 

 

Game Announcement: Stargazer

It’s been almost 2 years since we released a game (The Descent) as we’ve been busy building and managing eCommerce apps for some of our clients. Early in January 2017 we decided to play around with some puzzle game ideas and have been working on a gameplay mechanic prototype ever since. We opted for a different development methodology this time – play test game mechanics first on a crude prototype and focus on technology later. Years ago we would have started with a specific technology in mind and then try to fit gameplay into the technology code already in progress (bad approach).

The prototype has pretty good potential to morph into a game of its own so at this moment we would like to announce Stargazer – targeted for release by Q1 2017. It will be exclusive for Android initially.

Stay tuned for more news.

 

 

Migrating from Parse Push Notifications

Several months ago Parse took everyone by surprise and informed developers that they would shut down all of their services by the end of January 2017. While the news initially seemed like an April Fool’s joke similar to their public announcement of Parse Pigeon, we disregarded the migration message that appeared in on the screen every time you logged into your Parse Dashboard:

The Parse hosted service will be retired on January 28, 2017. If you are planning to migrate an app, you need to begin work as soon as possible.

However after countless logins into the Parse Dashboard the message kept appearing in the topmost banner, almost as if laughing at us, and gradually we began coming to terms with the harsh reality – Parse was indeed shutting down and we were (again) forced to migrate our apps to a different Push Notifications provider. This is not new to us – we’ve been dealing with these kinds of migrations since early 2015 when we migrated one of our apps from Urban Airship to Parse.

Back then we had a single Android app that used Urban Airship to receive push notifications from a web application written in Rails. The notifications workflow was simple – the app user would receive a push notification every time an order was placed by a customer through the YouOrderIt platform. The Android app was intended for an administrative audience such as restaurant managers and admin staff and the push notifications simply contained basic order details. We built the app in October 2014 and selected Urban Airship as the push platform of choice because it was free and because we refused to evaluate other options – Urban seemed like a tried and tested choice so we picked it.

And then sometime in January 2015 Urban Airship began charging monthly fees for their services. We had been all along in the beggar’s free tier, which they couldn’t support anymore (I don’t think any provider truly achieves high margins in this push notifications business) so we got an email from Urban Airship Support to the effect that we had to pay a $200 monthly fee going forward. We didn’t have that kind of money to spare so we had to switch to something else (free) immediately.

A friend recommended Parse so we started looking into it. The platform seemed solid, very well documented and their Dashboard was more user friendly than Urban Airship’s. They also had a beggar’s free tier throttled up to 30 requests per second (RPS) and we were confident we would fall beneath that threshold since our apps were not experiencing load above 30 RPS. We migrated the Android app to Parse and experienced good push notifications performance and decent reliability. After this experience we decided to use Parse for our complete app portfolio.

So a couple of months ago while mourning Parse’s looming demise we were faced with a tough question: migrate to what? We were back again to the drawing board just like in the pre-Urban Airship days. For a complete month we deliberated the idea of implementing our own platform based on Amazon SNS, and we even studied technical guides on how to migrate from Parse to Amazon SNS (gruesome process), but we didn’t wish to become a push notifications technology provider. That would have further derailed us from our core focus.

We realized that no matter what we wouldn’t be able to come up with a flawed assessment that identified a perfect, always-free and long-lived push notifications platform. The #1 push notifications platform today might be dead one year from now (even if it’s backed by Amazon, Google or even Facebook – ironically, before announcing their shut-down Parse had been engulfed by Facebook), and that’s just a fact of the brutality of technology. Even if it doesn’t die the platform’s creator might wake up annoyed one day and decide to start charging monthly fees, giving you only two viable options – pay or get the hell out and migrate.

Given the fast-changing technology landscape we are constantly subjected to, we have to be willing to switch platforms ever year if possible and be prepared for the changes entailed with these migrations. We decided to switch to the next best FREE choice that catered to our requirements. So when another colleague recommended OneSignal, we checked it out and after noticing it was free and had a decent breadth of features, we decided to migrate to that.

Android Setup

From an Android standpoint migrating to OneSignal only entails a couple of code changes. Besides the OneSignal SDK setup, generation of a Google Server API Key and Gradle changes mentioned in the OneSignal documentation, you just need to add the following initialization method. Make sure this is called in your main Application’s onCreate() method:

void initOneSignal() {
    OneSignal.startInit(this).
              setNotificationOpenedHandler(notificationsHandler).
              setNotificationReceivedHandler(notificationsHandler).
              init();
}

In this example we are using a custom notifications handler but this is optional. For instance if your app only needs to display the push notification using the built-in behavior a custom notifications handler is not required. However if you need your app to perform custom logic every time a notification is received or opened, we recommend using a notifications handler:

public static PushNotificationsHandler notificationsHandler;

If you decide to use a custom notifications handler make sure you override the notificationReceived() and notificationOpened() methods depending on your notification handling requirements.

public class PushNotificationsHandler implements OneSignal.NotificationOpenedHandler, OneSignal.NotificationReceivedHandler {

    public PushNotificationsHandler() {
        super();
    }

    @Override
    public void notificationReceived(OSNotification notification) {
        // Handle notification received event here
    }

    @Override
    public void notificationOpened(OSNotificationOpenResult result) {
        // Handle notification opened event here
    }

}

That’s all for the basic code changes. However these code changes might not be enough to fully migrate your app and platform to the new notifications service. Consider the following situation – you make your code changes in your Android app to migrate from Parse to OneSignal and push the changes to Google Play Developer Console. If you already have users using a previous version of your app (the codebase of the previous version uses Parse or some other platform), they might opt to NOT update the app (which contains the new OneSignal code changes) for a while. You will be faced with a situation where different users are running different versions of your app – one Parse enabled and the other OneSignal enabled, and you have to support them both for business continuity’s sake.

As part of your business, if you are sending notifications to apps directly from the Admin Dashboard, this is simple since you can manually target these two audiences separately. However if you have a complex platform where notifications are sent from a web application that sends notifications to devices using the OneSignal API, you need to write logic to handle these 2 separate paths – the path to send notifications to devices using Parse and the other path where you send notifications via OneSignal. However this code will be temporary, once all of your users update their apps to the newest version using OneSignal, you just need to handle the OneSignal path.

We faced this situation with one of our apps. We have an e-commerce app that can be used to place orders. It is currently used by thousands of customers and was originally Parse-enabled. With the Parse implementation, when users placed orders the Android app would send order details to our platform’s RESTful JSON API, including the Parse device token of the user’s device placing the order. This info would be saved in a database server-side for later use. If an order was confirmed, the web app would read the order details (including the device token), and send a push notification to that specific device token using the Parse API. This was the Parse path. To handle OneSignal we made one small code change in the Android app – the app would send the device’s OneSignal userId instead (also known as playerId) to our platform’s API when sending order details. Retrieving the userId is simple, but you have to make sure you retrieve it after OneSignal successfully initializes. Also the implementation is asynchronous so your Android app needs to handle this non-blocking behavior gracefully:

// Call this inside your Android app if you need to retrieve the device's userId for later use

OneSignal.idsAvailable(new OneSignal.IdsAvailableHandler() {
    @Override
    public void idsAvailable(String userId, String registrationId) {
        if (userId != null) {
            // Save userId or registrationId
        }
    }
});

When we introduced OneSignal we had to create a separate path server-side to handle the OneSignal-enabled apps. The server-side pseudocode to handle both paths ran something like this:

// The code below runs when the server needs to send a push notification to a device

Order order = getOrderById(orderId);

// Parse path
if (order.hasParseDeviceToken()) {
    String deviceToken = order.getDeviceToken();
    Notifications.sendPushUsingParse(deviceToken);
}
// OneSignal path
else if (order.hasOneSignalUserId()) {
    String userId = order.getUserId();
    Notifications.sendPushUsingOneSignal(userId);
}

Once all of your users update their Android apps to the latest version running OneSignal, you no longer need the first IF statement in the server-side code to handle the Parse path. You can remove it and your notifications platform will continue uninterrupted.