Our Journey about adapting AI into our game - Postmortem
Play our game here:
https://ldjam.com/events/ludum-dare/51/10-second-magic-paintbrush
Hello this is DGSpitzer, I’m the programmer from the team who made 10-Second Magic Artist in LLD 51. Today I want to introduce the tech pipeline and design process we used!
The Original Idea
When we saw the theme “Every 10 seconds”, a lot of different game ideas actually came up during our brainstorm session quite easily. However, most of the ideas are still a bit cliche, or maybe not funny/innovative/theme-oriented enough, as a result we arranged another brainstorm meeting the next day to filter up ideas and to get more discussion around the theme. We always want to find an innovative way to utilize cutting-edge technology into our game. As for myself, I’ve been trying out a lot of different approaches to do AI painting recently. Here is one of the animation video I made with Disco Diffusion AI:
https://www.youtube.com/watch?v=qM0BD2fzyBA
Among all of our ideas, an AI painting game design popped up on our idea ranking list eventually: Let the player scrabble following the NPC clients’ orders, and the NPCs’ demand will rapidly change every 10 seconds. We’ll use AI backend such as Stable Diffusion etc. to convert the player's input scrabble painting with img2img technology, alongside the text prompts created by the NPCs to generate a fancy artwork together! It would be a pretty good way to showcase the boost AI technology could have for human creators, and how it could make everybody’s imagination become true.
Initial Test
Will the result be good? We did a quick test during the brainstorm session, our team members drew one or two very rough sketch in couple seconds by following random prompt sentences being given:
I quickly convert those sketch use Stable Diffusion WebUI:
The result looks pretty promising to us, so we decide it’s a go!
Deployment
I had no experience of deploying a GPU backend or setting up a server - the only similar thing I did before is using Microsoft Playfab to create a simple online leaderboard / inventory system. Therefore it’s quite challenging for me to do it, especially in 72 hours! I dug into the Internet to see what I can do about it… Currently there are several ways to get a GPU server very easily , such as renting a GPU through vast.ai / AWS / Azure etc., or directly using an official API from DreamStudio or other AI drawing platform, in the end considering both my budget, customized ability and convenience, I decide to use banana dev to host a serverless GPU backend.
By default banana does give a one click deploy button for Stable Diffusion AI, however it only supports the txt2img function, it can’t achieve what we want to do in the development. Eventually I used this open source project as a base template to modify: https://github.com/sahil280114/stable-diffusion-inpainting-template/
By modifying it and deploying it through github to the serverless server, we can finally let AI convert! I faced some obstacles during the deployment process though, e.g. I used a customized template, seems like we need to change sanic==22.6.2 in requirement.txt during installation, otherwise the latest sanic version will crash the server side and create a failure deployment, also the log system in banana was not responsive somehow during the game jam, I need to use a temp beta log url for debugging. Each back and forth deployment will take me 1 hour to debug, kinda time consuming in my opinion (but fun to play with).
Unity Front End
For the front end, since the template only has a Python frontend, I need to write my own front end in Unity for sending and receiving data from our remote GPU. Fortunately it does support RestAPI with endpoint https://api.banana.dev/start/v4 The basic RestAPI from banana looks like this:
Additionally, the connection will be held for up to 50s waiting for the output from AI, if times out, I need to use another endpoint https://api.banana.dev/check/v4 with task Call ID from returned json to receive the output.
Normally the inference process from our GPU API is relatively fast (≈4s each output paintings). However, another issue is, our server will shut down if it is idle for a couple seconds, and the next player has to wait 50-70 seconds for the server to warm up and be available again. I then did a trick in our Unity front end, whenever the game starts and timer starts ticking, the game will keep sending an empty request to our API each 20s and let the server start warming up. By the time the players finish and submit their drawings, the API should be ready to use, and after the scene switch animation (5s), our players will get their AI generated artworks immediately in no time!
In the end, that’s how we create the smooth experience for dynamic AI generation in our game!