text-to-image plugin - A little suggestion about wrapper and input nodes
Hi! Maybe I found something for the t2i.
As far as I know... it uses flux.1 schnell with two encoders which are
- CLIP ViT/L (openai/clip-vit-large-patch14)
- T5xxl (google/t5-v1_1-xxl)
So the T5 is the biggest piece with up to 512 tokens monstrosity creating dense per-token map of vectors, capable of understanding context and was trained on text only. CLIP on other hand is 75 tokens long (77 with special starting and ending tokens) was trained vastly on content similar to SD, so it likes more a form of tags than natural language. And it generates a single pooled embedding for the whole in the beginning. Because of the architecture, the canvas precedes the information about it. Guess what, the CLIP overwhelms T5. Try to imagine it as a spiderweb nailed to the wall. The web is the T5 and nails are the CLIP. T5 says to the transformer "i want three apples in the left!" but CLIP was here before and already said "no, one big apple, on a table, you not changing it". Maybe if you say "three apples on a table" you'll be more lucky, because guidancescale multiplier doesn't help much with 4 steps in the schnell distro, it actually strengthen the gates that CLIP goes through... but that's a different story.
What I found is that the implementation doesn't expose something as prompt and prompt_2 parameters known in ComfyUI, which allows input the CLIP's and T5's nodes separately. With this separation trick the CLIP at least doesn't read the horrific 300 token description against the tags and simple phrases format from it's training. Then also T5 doesn't get confused with style tags and focuses only on the natural language it likes more.
Current solution is creating styles which exceed 75 tokens and leave the rest of the prompt for T5, but it reads the whole and gets fed with input vectors that strengthen it's result toward CLIP's canvas anyway, on top of being already capped by the CLIP's singular pooled vector.
So my suggestion is to expose the prompt_2 input node and leave both nodes free to use separately. Feeding two different encoders with the same input, where one is trained on short tags and second is a vast text encoder, only makes things worse.
This prompt_2 is baked into the architecture, so it doesn't need any complex crazy wrapping and exposing the parameter should be fine. Wrapper needs to extract second input as separate, have prompt_2 specified in the regex and apply to the iframe. On the server-side, API should get two separate fields from the JSON and route them separately to CLIP and T5 respectively. If the prompt_2 is null of unspecified, it backs to prompt one.
Sources:
https://arxiv.org/html/2507.09595v1#abstract1