Video generating LLMs - Conferring with other LLMs before generating the next scene
I was wondering if this either exists, or would be a useful addition to current video generation llm's (and any other tbh). I think it would be useful for a video generation LLM (like seedance), to confer with a storyteller LLM, a director LLM, a cinematography LLM, an acting LLM, etc, before it comes up with the next token. It could take the results of those suggestions, and add them into the context of the next token generation. Perhaps not for every token, but only ones symbolizing the end of a "scene", for example. This could lead to better camera angles, better acting, but story elements, plot points, etc. Do you think this could work? Or is it already being done? submitted by /u/BECOMING_A_TURTLE
Originally posted by u/BECOMING_A_TURTLE on r/ArtificialInteligence
1 reply
This is basically a mixture-of-experts approach applied to video gen, and it's a cool idea in theory but the latency cost would be brutal right now. Multiple LLM calls per scene boundary means you're multiplying inference time significantly, and video gen is already slow. Might make more sense as a preprocessing step, generate a full script/shotlist with a "director" LLM first, then feed that whole plan into the video model rather than conferring live.
Also worth noting, tools like Seedance 2.5 (https://klifgen.app/create-seedance-2-5) already lean on decent prompt understanding for coherent scenes, so the groundwork for what you're describing might already exist under the hood, just not exposed as separate conferring agents.