Well, they might be really shocked at the power levels of Goku and Broly. RWBY characters are used to their own kind of battles, and seeing the insane energy blasts and transformations in the Goku vs Broly fanfic could leave them in awe. They might start thinking about how they could incorporate some of those powerful moves into their own fighting styles.
Goku could escape by overpowering Broly. He might tap into his Super Saiyan forms, like going into Super Saiyan Blue or even Ultra Instinct if he's pushed far enough. His sheer power and fighting skills could allow him to break free from whatever restraints Broly has put on him.
Sure. Fanfiction writers have a lot of creative freedom when it comes to Goku and Female Broly. In some fanfictions, they might be in a romantic relationship, which is an interesting take considering their Saiyan heritage and the way they both value strength. However, not all fanfictions take this route. Some might explore a more platonic friendship where they respect each other's power and learn from one another. You can usually find a variety of these stories by using the search function on major fanfiction platforms.
Well, if they were brothers in a fanfic, it could change how they interact with the other characters. Maybe they'd have a deeper understanding and fewer power struggles between them. Also, their family background and upbringing could be explored in new and interesting ways.
It's a bit hard to say for sure. Both Vegeta and Goku have their moments of strength. Goku has some unique powers and techniques that give him an edge at times, but Vegeta is no slouch either and constantly works to catch up and surpass Goku.
Generally, Goku still holds the edge over Vegeta. Their power levels fluctuate, but Goku often achieves higher peaks of strength and unlocks more powerful forms first.