{"author_link":"\/users\/dgspitzer","author_name":"DGSpitzer","author_uid":"dgspitzer","comments":[],"epoch":1666374788,"event":"LD51","format":"md","ldjam_node_id":310803,"likes":13,"metadata":{"p_key":"169527","p_author":"DGSpitzer","p_authorkey":"1014379","p_urlkey":"397302","p_title":"Our Journey about adapting AI into our game - Postmortem","p_cat":"LDJam ","p_event":"LD51","p_time":"1666374788","p_likes":"13","p_comments":"0","p_status":"WAYBACK","us_key":"1014379","us_name":"DGSpitzer","us_username":"dgspitzer","event_start":"1664496000","event_key":"114","event_name":"Ludum Dare 51"},"node":{"_collation":{"body_sanitizer":"TextUtils::SanitizeHTML via existing importer","event":"LD51","removed_author":false},"_superparent":296586,"_trust":3,"author":14379,"body":"## Play our game here:\n## https:\/\/ldjam.com\/events\/ludum-dare\/51\/10-second-magic-paintbrush\n\n![51c8f.jpg](\/\/\/raw\/b28\/3\/z\/5358e.jpg)\n\nHello this is DGSpitzer, I\u2019m the programmer from the team who made 10-Second Magic Artist in LLD 51. Today I want to introduce the tech pipeline and design process we used!\n\n\n## The Original Idea\nWhen we saw the theme \u201cEvery 10 seconds\u201d, a lot of different game ideas actually came up during our brainstorm session quite easily. However, most of the ideas are still a bit cliche, or maybe not funny\/innovative\/theme-oriented enough, as a result we arranged another brainstorm meeting the next day to filter up ideas and to get more discussion around the theme.\nWe always want to find an innovative way to utilize cutting-edge technology into our game. As for myself, I\u2019ve been trying out a lot of different approaches to do AI painting recently.\nHere is one of the animation video I made with Disco Diffusion AI:\n\nhttps:\/\/www.youtube.com\/watch?v=qM0BD2fzyBA\n\nAmong all of our ideas, an AI painting game design popped up on our idea ranking list eventually:\nLet the player scrabble following the NPC clients\u2019 orders, and the NPCs\u2019 demand will rapidly change every 10 seconds. \nWe\u2019ll use AI backend such as Stable Diffusion etc. to convert the player's input scrabble painting with img2img technology, alongside the text prompts created by the NPCs to generate a fancy artwork together!\nIt would be a pretty good way to showcase the boost AI technology could have for human creators, and how it could make everybody\u2019s imagination become true.\n\n![523df.png](\/\/\/raw\/b28\/3\/z\/53596.png)\n\n## Initial Test\n\nWill the result be good? We did a quick test during the brainstorm session, our team members drew one or two very rough sketch in couple seconds by following random prompt sentences being given:\n\nI quickly convert those sketch use Stable Diffusion WebUI:\n\n![WeChat Image_20221001125845.png](\/\/\/raw\/b28\/3\/z\/5358f.png)\n\nThe result looks pretty promising to us, so we decide it\u2019s a go!\n\n## Deployment\n\nI had no experience of deploying a GPU backend or setting up a server - the only similar thing I did before is using Microsoft Playfab to create a simple online leaderboard \/ inventory system. Therefore it\u2019s quite challenging for me to do it, especially in 72 hours! I dug into the Internet to see what I can do about it\u2026 Currently there are several ways to get a GPU server very easily\n, such as renting a GPU through vast.ai \/ AWS \/ Azure etc., or directly using an official API from DreamStudio or other AI drawing platform, in the end considering both my budget, customized ability and convenience, I decide to use banana dev to host a serverless GPU backend.\n\n![WeChat Image_20221021134246.png](\/\/\/raw\/b28\/3\/z\/53592.png)\n\nBy default banana does give a one click deploy button for Stable Diffusion AI, however it only supports the txt2img function, it can\u2019t achieve what we want to do in the development. Eventually I used this open source project as a base template to modify: https:\/\/github.com\/sahil280114\/stable-diffusion-inpainting-template\/\n\nBy modifying it and deploying it through github to the serverless server, we can finally let AI convert! I faced some obstacles during the deployment process though, e.g. I used a customized template, seems like we need to change sanic==22.6.2 in requirement.txt during installation, otherwise the latest sanic version will crash the server side and create a failure deployment, also the log system in banana was not responsive somehow during the game jam, I need to use a temp beta log url for debugging. Each back and forth deployment will take me 1 hour to debug, kinda time consuming in my opinion (but fun to play with).\n\n![WeChat Image_20221001225303.png](\/\/\/raw\/b28\/3\/z\/53593.png)\n\n## Unity Front End\n\nFor the front end, since the template only has a Python frontend, I need to write my own front end in Unity for sending and receiving data from our remote GPU. Fortunately it does support RestAPI with endpoint https:\/\/api.banana.dev\/start\/v4 \nThe basic RestAPI from banana looks like this:\n\n![1.png](\/\/\/raw\/b28\/3\/z\/53594.png)\n\nAdditionally, the connection will be held for up to 50s waiting for the output from AI, if times out, I need to use another endpoint https:\/\/api.banana.dev\/check\/v4 with task Call ID from returned json to receive the output.\n\nNormally the inference process from our GPU API is relatively fast (\u22484s each output paintings). However, another issue is, our server will shut down if it is idle for a couple seconds, and the next player has to wait 50-70 seconds for the server to warm up and be available again. \nI then did a trick in our Unity front end, whenever the game starts and timer starts ticking, the game will keep sending an empty request to our API each 20s and let the server start warming up. By the time the players finish and submit their drawings, the API should be ready to use, and after the scene switch animation (5s), our players will get their AI generated artworks immediately in no time!\n\n![AIGalleryPainting-2022-10-05_16-18-56-s.png](\/\/\/raw\/b28\/3\/z\/53595.png)\n\nIn the end, that\u2019s how we create the smooth experience for dynamic AI generation in our game!\n\n## Would you like to know more? You can try the game out\n\n## https:\/\/ldjam.com\/events\/ludum-dare\/51\/10-second-magic-paintbrush\n","comments":2,"comments-timestamp":"2022-10-21T17:57:32Z","created":"2022-10-21T16:16:52Z","files":[],"files-timestamp":0,"id":310803,"love":13,"love-timestamp":"2022-10-22T00:13:10Z","meta":[],"modified":"2022-10-22T00:13:10Z","name":"Our Journey about adapting AI into our game - Postmortem","node-timestamp":"2022-10-21T17:53:08Z","parent":305529,"parents":[1,5,9,296586,305529],"path":"\/events\/ludum-dare\/51\/10-second-magic-paintbrush\/our-journey-about-adapting-ai-into-our-game-postmortem","published":"2022-10-21T17:53:08Z","scope":"public","slug":"our-journey-about-adapting-ai-into-our-game-postmortem","subsubtype":"","subtype":"","type":"post","version":969763},"node_metadata":{"n_key":"310803","n_urlkey":"397302","n_parent":"305529","n_path":"\/events\/ludum-dare\/51\/10-second-magic-paintbrush\/our-journey-about-adapting-ai-into-our-game-postmortem","n_slug":"our-journey-about-adapting-ai-in","n_type":"post","n_subtype":"","n_subsubtype":"","n_author":"14379","n_created":"1666369012","n_modified":"1666397590","n_version":"969763","n_status":"WAYBACK"},"source_url":"https:\/\/ldjam.com\/events\/ludum-dare\/51\/10-second-magic-paintbrush\/our-journey-about-adapting-ai-into-our-game-postmortem","text":"## Play our game here:\n## https:\/\/ldjam.com\/events\/ludum-dare\/51\/10-second-magic-paintbrush\n\n![51c8f.jpg](\/\/\/raw\/b28\/3\/z\/5358e.jpg)\n\nHello this is DGSpitzer, I\u2019m the programmer from the team who made 10-Second Magic Artist in LLD 51. Today I want to introduce the tech pipeline and design process we used!\n\n\n## The Original Idea\nWhen we saw the theme \u201cEvery 10 seconds\u201d, a lot of different game ideas actually came up during our brainstorm session quite easily. However, most of the ideas are still a bit cliche, or maybe not funny\/innovative\/theme-oriented enough, as a result we arranged another brainstorm meeting the next day to filter up ideas and to get more discussion around the theme.\nWe always want to find an innovative way to utilize cutting-edge technology into our game. As for myself, I\u2019ve been trying out a lot of different approaches to do AI painting recently.\nHere is one of the animation video I made with Disco Diffusion AI:\n\nhttps:\/\/www.youtube.com\/watch?v=qM0BD2fzyBA\n\nAmong all of our ideas, an AI painting game design popped up on our idea ranking list eventually:\nLet the player scrabble following the NPC clients\u2019 orders, and the NPCs\u2019 demand will rapidly change every 10 seconds. \nWe\u2019ll use AI backend such as Stable Diffusion etc. to convert the player's input scrabble painting with img2img technology, alongside the text prompts created by the NPCs to generate a fancy artwork together!\nIt would be a pretty good way to showcase the boost AI technology could have for human creators, and how it could make everybody\u2019s imagination become true.\n\n![523df.png](\/\/\/raw\/b28\/3\/z\/53596.png)\n\n## Initial Test\n\nWill the result be good? We did a quick test during the brainstorm session, our team members drew one or two very rough sketch in couple seconds by following random prompt sentences being given:\n\nI quickly convert those sketch use Stable Diffusion WebUI:\n\n![WeChat Image_20221001125845.png](\/\/\/raw\/b28\/3\/z\/5358f.png)\n\nThe result looks pretty promising to us, so we decide it\u2019s a go!\n\n## Deployment\n\nI had no experience of deploying a GPU backend or setting up a server - the only similar thing I did before is using Microsoft Playfab to create a simple online leaderboard \/ inventory system. Therefore it\u2019s quite challenging for me to do it, especially in 72 hours! I dug into the Internet to see what I can do about it\u2026 Currently there are several ways to get a GPU server very easily\n, such as renting a GPU through vast.ai \/ AWS \/ Azure etc., or directly using an official API from DreamStudio or other AI drawing platform, in the end considering both my budget, customized ability and convenience, I decide to use banana dev to host a serverless GPU backend.\n\n![WeChat Image_20221021134246.png](\/\/\/raw\/b28\/3\/z\/53592.png)\n\nBy default banana does give a one click deploy button for Stable Diffusion AI, however it only supports the txt2img function, it can\u2019t achieve what we want to do in the development. Eventually I used this open source project as a base template to modify: https:\/\/github.com\/sahil280114\/stable-diffusion-inpainting-template\/\n\nBy modifying it and deploying it through github to the serverless server, we can finally let AI convert! I faced some obstacles during the deployment process though, e.g. I used a customized template, seems like we need to change sanic==22.6.2 in requirement.txt during installation, otherwise the latest sanic version will crash the server side and create a failure deployment, also the log system in banana was not responsive somehow during the game jam, I need to use a temp beta log url for debugging. Each back and forth deployment will take me 1 hour to debug, kinda time consuming in my opinion (but fun to play with).\n\n![WeChat Image_20221001225303.png](\/\/\/raw\/b28\/3\/z\/53593.png)\n\n## Unity Front End\n\nFor the front end, since the template only has a Python frontend, I need to write my own front end in Unity for sending and receiving data from our remote GPU. Fortunately it does support RestAPI with endpoint https:\/\/api.banana.dev\/start\/v4 \nThe basic RestAPI from banana looks like this:\n\n![1.png](\/\/\/raw\/b28\/3\/z\/53594.png)\n\nAdditionally, the connection will be held for up to 50s waiting for the output from AI, if times out, I need to use another endpoint https:\/\/api.banana.dev\/check\/v4 with task Call ID from returned json to receive the output.\n\nNormally the inference process from our GPU API is relatively fast (\u22484s each output paintings). However, another issue is, our server will shut down if it is idle for a couple seconds, and the next player has to wait 50-70 seconds for the server to warm up and be available again. \nI then did a trick in our Unity front end, whenever the game starts and timer starts ticking, the game will keep sending an empty request to our API each 20s and let the server start warming up. By the time the players finish and submit their drawings, the API should be ready to use, and after the scene switch animation (5s), our players will get their AI generated artworks immediately in no time!\n\n![AIGalleryPainting-2022-10-05_16-18-56-s.png](\/\/\/raw\/b28\/3\/z\/53595.png)\n\nIn the end, that\u2019s how we create the smooth experience for dynamic AI generation in our game!\n\n## Would you like to know more? You can try the game out\n\n## https:\/\/ldjam.com\/events\/ludum-dare\/51\/10-second-magic-paintbrush\n","title":"Our Journey about adapting AI into our game - Postmortem","wayback_source":[]}