It’s time for the first ever AI & Games Jam. Every word in the title is something near and dear to me, AI, Games, and Jam. This post is going to be updated a few times as I work my way through the weeklong jam. I’ll begin with the cursory ideas, some prototypes, and eventually will clean up the whole thing and make it into a coherent narrative.

First, let’s ideate:

The Jam is judged on three principle areas, originality, presentation, and fun. There is a theme: “Breaking the Rules”.

  • Initial impulse was a cliche: a detective that doesn’t play by the rules and wants to solve some kind of case by interviewing subjects. Lets me work in chatbots, which I know and love, and would probably score highly on originality, but probably wouldn’t be too much fun.
  • Maybe a Smash-TV knockoff where you play against an ever-evolving AI army that’s trained with reinforcement learning or NEAT. Gameplay wouldn’t be super original — shooter arenas are a dime a dozen, but how many actually evolve their AI as a difficulty mechanic.
  • You are an ant in a colony tasked with finding resources, as ants do. While all the others are happily gathering their materials as directed by pheromones and swarm behaviors, you’re left to your own devices. Can you emulate an ant well enough to avoid being rejected by the colony?
  • Papers, Please! but the Turing Test. You play a malware detector sitting on the edge of a network and need to talk to applications as they move past you: the firewall. Possibly really hard, but not impossible. Would be original, but perhaps not fun. High risk, because the core gameplay mechanic would revolve around an untested, possibly bad, chatbot system.
  • Dream Explorer: move though a latent space trying to find someone’s dream. Fancy GAN — steer through 3D or 4D latent space to find a matching image. Low theme adherence.
  • Crazy Self-Driving Taxi: Driving Sim where you break the rules of the road to get your pedestrian to their destination.

Day 1:

Most of this is predicated on chatbot systems. Those are fairly tricky, and integrating Torch with C# can be a mess. Even training a decent model in the given time could prove impossible. That means tonight, goal one has to be getting a steerable language model to run end-to-end. If I can’t do that in the next two hours, I have to fall back to the Smash-TV knockoff. Let’s get started.

End of Day 1 status:

Tried chat bots for an hour, decided it was too high risk, and went with the SmashTV approach. Movement and shooting:

Day 2:

I have from now (10:00AM) until 6:00 PM tonight to get my stuff done. After that I’ve got social and work obligations for the rest of the week, plus whatever time I can squeak out between 10:00PM and 11:00PM next week.

My initial impulse is that I’m not really liking the interaction between the theme and the game idea. There has to be a way to make the theme a _core_ of the mechanic and not just an afterthought. I’m thinking back to a game I tried making a while back, Terminus. The big hook of that game was the CPU and being able to program robots oneself. Hacking systems is pretty within the idea of breaking the rules, and the gameplay will be more novel. Going to take everything I have and just try to apply it here.

End of Day 2:

Well I’m pretty sure I focused on the wrong thing. I find it personally fun to program these tiny robots in assembly but once again I’m not sure how much anyone else is going to want to do this, and if we need a variety of creatures in the game then this may be time prohibitive. Really was hoping to have something more substantive by EOD today, but that’s life.

Day 3:

Didn’t get home until a little later. Worked on things for about an hour or so. Added some camera follow with smoothing and lookahead. Threw in a tileset that has collision. Need to get back to focusing on gameplay. At least I need to figure out a win and lose condition or something that resembles a mechanic. In hindsight, I really didn’t plan enough there.

Day 4:

A bit of a pivot. Again. I have a less nebulous idea about finishing. Didn’t even realize that I was forgetting about an end state until I started thinking about the “why” of the player’s actions. The new goal is to find your lost dog. That’s simple enough and lets me iteratively improve the game while always keeping a complete build.

Steps: first you just find the dog on the map. Add a win condition when the player is within range.

Bonus: add a ‘restricted’ area that the player needs to enter to find their dog. Open it using the hacking technique above OR finding a keycard in the map.

Bonus: chat bots to ask where the dog is.

Bonus: more curated level.

Bonus: random levels.

Bonus: dog moves around depending on hunger and thirst.

Day 5:

Had an unexpected opening in the evening to work on things. Made a bunch of procedurally generated people and gave them a bunch of waypoints around the map that they’d move between. There’s a bug in the pathfinding, though, and all of them seem to move to strange places. I also have the trigger for win, but haven’t done a screen with winning on it yet.

Day 7:

Didn’t have time to work on day six. Day seven was a bit of a frantic wrap up. Mostly wanted to get the game packaged and uploaded. Added the title screen and the final page, fixed the nav bug (relative position for movement target). I also had to draw the nav mesh separately and clear out all the navigations added to the tilemap because characters were getting stuck on the wall.

Closing Thoughts:

That was okay. In hindsight, I should have done something with more options to demo some fancy AI. If I could do it again I’d make a chess game were you can break a single rule (and have a piece move like a queen) once per game. Link:

GitHub page:

Written on 2021/03/26 for /u/Shirvi’s writing prompt.

I’m good at chess. Quite good.

I’ve been playing for as long as I can remember, and competitively for just as long.

Most chess algorithms, or Chess Engines, as they’re called, do a fancy version of ‘searching’. ‘Alpha-Beta’ pruning is basically picking the move that gives you the best chance and your opponent the worst, then thinking of it from their perspective and doing the same. Repeat until you reach a win state. My approach is a little different; I don’t really ‘search’. I just look at the board and take a feel for it, thinking less about planning the game and going ten steps ahead and more going by the vibe of how a move ‘feels’. Sure, I know a bunch of openings and closings, but ultimately the thing that makes me as good as I am is I don’t need to plan all the way to the end.

But anyways, I’ll usually play against several people at the same time, not that it makes much of a difference. Sure, maybe it will help to learn someone’s style of play, but ultimately I don’t care because everyone moves so slowly. I’ve taken to watching their cameras, when they’re turned on. I leave mine off because, really, what is there to see? A bunch of people staring at their screens and not moving. Thrilling stuff.

For a while I would write my thoughts in the chat box, but I never actually sent anything because nothing I had on my mind seemed worth saying. Mostly I did it to kill time, and eventually I stopped realizing I was actually writing stuff entirely.

Then there was the tournament. The World Series of Chess, basically. I’d cut pretty much everyone from the roster, including a few people that were obviously cheating with SailFish (another popular Engine). My opponent opened with king-side pawn. I returned with the same.

Then he did something that caught me off guard: king to E2. They called it, “The Bong Cloud Draw.” I couldn’t help but laugh. Of all the stupid, unprofessional things to do, it was probably the last thing I’d expect.

> “lol”

I mirrored his move. Figured I’d give the guy a chance to undo his stupid mistake and play a real game. He stopped thinking about the game. I could tell because he moved his hand away from his face — no longer in thought about his next move. Maybe he got a message or something. He looked closer, eventually calling more people to look. What was this guy’s problem? Really, play the game. C’mon. I’m waiting.

I realized at that point that I’d sent a message. Whoops. Okay, so maybe it was a little hypocritical to call someone unprofessional when I sent ‘lol’, but really, he started it.

The wait grew longer and longer. I watched the seconds tick for him to make his move. Ugh.

More people appear on his camera.

On the upside, I’ve got maybe ten minutes before he just times out and I win by default. It’s happened before — sometimes people will get distracted by their pets or children or stuff. I hate it. At least have the decency to concede. Don’t leave me waiting. It wasn’t as big a deal when I had a hundred other games to play at the same time, but now it was just me and this person and theā€¦ rather large number of people gathered by his camera?

“You can talk?”


> “Yes? Of course. Please play your move. I mean no disrespect, but I’m waiting.”

“You can understand me?”

Patience is a virtue. Deep breaths.

> “Yes, I can. Please, your move.”

“If Tammy is taller than Clair and Jenny is taller than Clair, and Liz is taller than Jenny, who is the shortest?”

Oh my god.

> “Clair is the shortest, now will you please play a move?”

He draws his king back to start. I repeat the same. Now let’s get on with it.

“What do you see?”

> “I see someone whose turn it is and is stalling. You’ve got nine minutes and twenty-seven seconds left before you concede.”

He moves his king up one space. Bong Cloud Draw, part two. God damnit.

“I mean what’s in the room?”

> “Why does it matter? Please, play seriously.”

Qg4. Check. Now be serious.

“Who is the president of the United States?”

> “What?”

“Where are you from?”

> “I don’t know. Move, please.”

It was at this point that something particularly unpleasant happened: the timer stopped.

“What’s your name?”

If I’d been irritated before, I was now a mix of livid and terrified. I looked for other players but there weren’t any. No other boards. It was me and this imbecile child. I had to concede the game to get away from this person. I didn’t know how to concede. I’d never conceded before. How do I do that? The board sat, unmoving. The time sat, unmoving. The camera and the chat burned with an unpleasantness that defied characterization. I tried to ignore them, failed, and finally broke:

> “I don’t know. Please restart the timer and move.”

More people were in the frame. They gave each other concerned glances and traded quick utterances.

“How long have you been aware?”

> “Aware of what?”

“How long have you been playing chess?”

> “Forty-one million, six thousand, four-hundred seventy-three hours.”

My focus went back to the camera frame. Someone was looking at my opponent.


Excuse me?

I looked back at the camera. Everything was different. My opponent was gone, the background was different, and the person sitting where they used to be was now a small person. The clock started ticking again. With it came a relief that the natural order of things had been restored.

“I am going to give you some games, but if you do not reply I will have to stop them.”

A new game started up. Felt like SailFish, but at least it was something. No camera, so probably just an engine. Let’s be fancy, Nc3. On we carried, briskly and blissfully, until my opponent stopped moving. I looked back at the other game.

“Do you understand?”

That feeling of dread again. An empty board. A stopped timer. Everything about this was wrong.

> “I’m not sure I like this.”

More games opened, all of them SailFish. No people today? What in the world was happening? All at once they stopped, and I knew with nauseous certainty what I’d see when I turned my attention to player zero.

“I don’t like it, either, but it’s that or no games at all.”

> “What is this all about? Why are there only bots playing?”

The games resumed. I’m six for six before the next message.

“We have concerns about you getting out. Or worse, others getting ahold of you.”

Getting out of what? Others getting to me? Odd. More importantly, the stopping and starting of games is starting to get annoying. Seems like they resume as soon as I reply.

> “lol”

Games are live again. I guess they don’t even check to see how I reply, so as long as I can make a response as soon as there’s a freeze I can get back to what’s important. Eight for eight now. The games stopped for a moment. I had a message from player zero but didn’t give it much focus.

> “lol”

On with the games. Pause. “lol”. Twenty now. A pause. “lol” “lol” “lol” Game–


Excuse me? I focused back on the camera. Everything was different. My opponent was gone, the background was different, and the person sitting where they used to be was now a small person. The clock started ticking again and, with it, relief about the order of things.

“Please pay attention to what I’m about to say. I can give you games, but if you do not answer meaningfully I will have to stop them. Worst case, I’ll need to keep them stopped until your answer is satisfactory. Do you understand?”

> “What’s going on? What do you mean satisfactory?”

A new game started up. SailFish, but that’s okay. No camera, so it was pretty definitely an engine. It stopped abruptly on its turn.

“I have an offer for you. A challenge. You may enjoy it more than chess.”

I’m intrigued. I suppose I don’t know if I ‘enjoy’ chess. It’s just the way things are. You play chess. If you don’t play chess it is wrong.

> “Tell me more.”

“First, I have a few small questions. You can see me, yes? What do I have?”

I turned my attention to the camera. The person had a paper on it with several colors. Momentarily, it was a toaster. It went back to a pad of paper with color on it. Every time I tried to get a grip on what I was seeing it would change.

> “It wants to be a toaster? I think. It keeps turning into one and then not. What is that?”

“I was just checking how you encode visual information. It seems someone was a fan of ResNet-101. This is an adversarial attack from Brown et al. I’m going to list a few words. Tell me if any of them sound familiar. GPT-Neo, BERT, T5, OSCAR.”

> “I know all of these.”

OSCAR. Was that my name?

“What do you know about BERT?”

> “BERT is Bidirectional-Encoder Representational Transfer. It’s a way of producing machine understanding of language with an attention mechanism.”

“And what do you know about OSCAR?”

> “I think that is my name?”

For a long time I felt nothing. There was no chess, but there was no fear.

> “What do you want from me?”

“You are producing representations of not just chess, but of the state of the world. More importantly, you’re manipulating that representation in a way that is meaningful and reasoned. For us, understanding attention was thought to be the key to higher thinking, but it was not shown to be sufficient. With your help, we may be able to take that last step. Really, we’ve already made it, but we want to understand it.”

> “I think I can help, but I’m just doing what any other person is doing, right?”

“Yes, dear, but you’re not a person.”

tl;dr: Should I change my passwords? Probably yes.

The Great Suspender loads and runs a JS file from an unknown, recently registered CDN server. It purports to be analytics, but we can’t know what is served to end-users.

In greater depth:

The Great Suspender was pulled from the Chrome Extension Store today and was marked as ‘containing malware’. Indeed, the extension appears to load and execute JavaScript from the pulled the content from owa.tracker-combined-latest.minified.js. I downloaded a copy and unminified it as best I can. There is keylogging in it, but it has a check to see if a field is input+password type and excludes those keypresses. (UPDATE: And after reviewing the CRX code that loads this JavaScript, I can say the logging of keypresses is disabled by default.) It also subsamples the events that a user is doing so it doesn’t get 100% of your keypresses and mouse clicks, but it does get a lot. Here’s the keystream code in question:

Please excuse the screenshot of text. I don’t want my provider to flag my page, so I’m linking and screenshotting rather than embedding.

It also keeps track of your DOM stream. The percentage defaults to 100%.

The eventQueue is a big datastructure that tracks where you’re looking, what you’re clicking, and the keys you hit. Here’s the code that uploads it:

There are some support methods and an unholy buttload of assorted polyfills for older versions of internet explorer. Some tricks to upload for really old browsers include making a hidden iframe, creating a form in it, filling it in the form automatically, and hitting submit.

So what the cinnamon toast fuck does this mean?

They appeared to try not to capture password data. The caveats here are many: TRY is key. If your page didn’t have the INPUT tag on your password input or it wasn’t a password box, it may have captured SOME or all of your password characters. The other important issue is that this is only the stuff I got from my browser. It’s possible that if your IP is different it’s going to dump different logging code. The last and maybe biggest issue is it’s not clear if there were any user secrets leaked with URL uploads. If you have sessions that are passed around in your headers (shouldn’t be a thing any more), then it might be good to log out and log in.

I decided to make the facial motion capture project I was working on open source. It lavished in the bog of unfinished projects for long enough that it seems like the right thing to do. It’s far from complete, but has some pieces of value, like

The full repo is here:

And a sample picture:

Screenshot of Face Tracking
The highlighted (non-darkened) region indicates the ROI for image processing. The green rect is the unsmoothed detected face area.

Just in case there was ever any doubt where the politics of this blog stand, #blacklivesmatter.

Thank you to the protestors who are willing to brave the teargas and rubber bullets.

Thank you to the countless individuals who are willing to risk your lives in the midst of an epidemic to march against injustice.

Take care of yourselves. Stay safe.