ok so at the time i was really into screen studio which lets you screen record these like really beautiful videos with padding and gradients and borders and auto-zooming based on mouse clicks, then i thought a lot about video editing bc my flow was screen studio video -> imovie + music -> post on twitter and that got me thinking about video editing and that got me thinking about image editing and i was like we should just be able to edit images with natural language. so i built this scrappy v1 where you upload an image, describe the edit, i prompt upscale and then send that over to a model on replicate and return the edited image, almost like a proto-4o. i posted it on x and it did pretty well. then i added simple video gen to the site using replicate again, and then i used this sdk to create a full blown video editing ui, the idea being that i would add transitions, have templates, private "draft" links, etc. so that founders could "streamline" their process for creating video demos. then i shipped this version that would let you upload a video and add music to it, and this grew into the last version which was like a web-based version of screen studio. so originally i wanted a tool to edit images w natural language but the base models just weren't good enough yet, and so i moved into trying to solve a problem i actually had, sharing private draft links for video demos, quickly adding music, making cool transitions easily, etc. but eventually i just got bored with this problem.

omg im remembering how fun this was, basically this was a time when i was really obsessed with langchain - i was just playing around a lot with their js and py sdks and i came across these generative agent classes, and i has just read this paper on generative agents so in a day i built this streamlit app (backend doesn't work anymore so just linking twt) that would let you spin up a simulation between multiple agents, where each "agent" was a generative agent abstraction on top of a base instruct model. then i realized that the generative agent classes only existed in the py sdk?? turns out langchain at the time (i’m not sure of their state now) was bifurcated between the js and py abstractions, where the py lib was always ahead of the js one and they were never at parity because the py sdk was developed first, and so i TRANSLATED the generative agent abstractions from py to ts because i wanted to build a js web game with these agents and there was no way to do so since they didn’t exist in the js/ts sdk LMAO a langchain team member even gave me a s/o on x for this lol. then i used my translated abstractions in the js sdk to start working on a web game where the game engine was phaser and i would connect the characters to the generative agents and have them interact with each other and the environment. i got both parts to work independtly in the web "game", i got the generative agents to work, and a very rough phaser game to work (with tilemaps and everything, etc.) but connecting these two proved to be a lot more challenging than i thought...