I see now that that is the step from the lip formed by your old transom. I should have read your description better. I’d be inclined to layer in 2 or 3 more layers of glass to bring the hollow up to level with the inside skin. It would be better than any other filler.When you tab new glass into old, it’s best to grind a taper into the old glass. Twelve to one is the generally accepted ration, so of your skin was 1/4” thick, you would taper back 12/4” or 3”. Then you layer in your glass so that it creates a level transition. There are two schools of thought, bigger pieces first, or smaller first. I now subscribe to the bigger pieces first, because you won’t be grinding through your full size pieces of glass to get a level surface. In something structural, like a transom, you would then add full layers that lap I to the hull sides over your level skin.Your transom will be fine. In fact by building out the hollow, you would be creating an inside out version of what I described.
Yeah like they said, if you round over the inside corners it should sit in a little better. I would start there and then see how it looks. I wouldn't want to grind out too much of the new glass since it is essentially holding it all in. Those three extra layers won't do you any good if you grind them out right at the corners. Also, not sure if you are using poly or epoxy but it may be worth the extra $$ for some epoxy to fill the gaps that are left since it will have more structural strength than the poly.
Take string and stretch it across the outside of the transom - is it a flat line or a bowed line?