Welcome aboard BK.
You'll need to cut the front liner out. Figure you'll need to have about 18" of room, cleaned and ground down in front of the transom, so you get a good, strong lamination of the new transom to boat. You can remove the cap screws that tie the cap into the hull, out along about 6 or 8 feet from the transom and you can pop the cap loose and pry it up so you don't have to cut it. You may have to cut back any liner on the hull sides too. That being said, you'll also have to carefully cut about 18" of stringers out - if you are careful, you can re-glass them back in if they're in good shape. Then grind the inside of the transom and the inside of the hull, both the sides and hull bottom inside the boat, out about 18", until nice and smooth. Make sure you grind out the corners along the hull sides and bottom.
The usual lamination schedule is 3 layers of 1708 across the newly ground transom and hull area and the overlap should be 12" on the first layer, 8" on 2nd and 4" on 3rd. Best way to get the glass ready is to take a 1x2 and clamp it to the top of the hullsides across the boat above the transom (under the propped up cap). Then clamp the 3 layers of cloth to the 1x2 and let the layers fall down into the boat (make sure you added the 12, 8 and 4" overlap lengths when you rough cut it. Flip the top 2 layers back out over the 1x2 and then using scissors, cut and pleat the first layer of glass in the corners - using a sharpie mark where the 12" overlay ends on the hull sides and bottom, this way you'll know how far out to wet the hull with resin. Flip the next layer into the boat and dry cut it in like the first but this layer should be 4" shorter than the first. Finally trim the 3rd layer. Flip all 3 layers back out of the boat over the 1x2.
Part of the lamination schedule is to round out all the corners - we do this because 1708 doesn't like to traverse sharp edges. You do this by mixing up a batch of thickened resin - thicken with chopped/milled fiberglass fibers, and/or wood flour and/or cabosil, usually it's a combo of several of these to get the thickness you want, usually like a peanut butter stiffness. Round the corners out by taking a wooden spoon or something that you rounded out and smear the thickened resin into the corner and then drag the spoon rounded edge along the corner to round it out and remove any excess. Try to be neat with this so you don't have much grinding to do. Let that kick. You'll probably end up with several size spoon devices through the build. Start with the smallest radius that you can to round out the corners. It's easy to trim'shape the spoon with a grinder.
Another thing you want to think about is you'll be repeating the 3 layers of 1708 on the inside of the transom, after clamping in the coosa, to laminate the coosa to the hullsides and bottom - all of the layers add thickness to the transom area, which unfortunately moves the low point of the hull forward at least a foot, which means that water won't be able to drain out the garboard plug anymore. I discovered this when I rebuilt my 170. So I always have an inch or so of water in the hull that I have to sponge out. Here is the fix for that - these links explain the problem and the last one shows a competed new drain:
Here is a link to where I explain the thought of the PVC pipe
http://classicaquasport.com/smf/index.php?topic=13148.msg133124#msg133124Here is a link to where I show what is actual problem
http://classicaquasport.com/smf/index.php?topic=13148.msg133556#msg133556Here is a link to Hawgleg's rebuild that shows the PVC pipe installation to keep a drain at the new low point of the boat
http://classicaquasport.com/smf/index.php?topic=13148.msg134262#msg134262So, using the thickened resin, glue the 12-18" pipe into the hull bottom while doing your corner rounding.
After the fillets of thickened resin harden, it's time to laminate the 3 layers of 1708 onto the transom, hull sides and hull inside bottom. Mix up enough resin that you can laminate all 3 layers of 1708 wet on wet. If using polyester don't catalyze it too hot and if epoxy, mix per instructions. After mixing the resin you have to get it out of the bucket and into a paint tray so it doesn't harden quickly. Roll the transom with resin, flip the first 1708, hard roll the cloth into the corners and make sure it is flat across the transom, add more resin to the cloth and flip the 2nd layer in hard rolling the resin out and the cloth into the corners nice and smooth. Add moire resin and flip the final layer of cloth into the boat and hard roll it everywhere. Make sure you hard roll into all the corners. Any excess resin will typically run down and pool along the transom hull bottom so don't over saturate the cloth.
Let that harden and then clean it up with a light grind. Then dry fit the coosa, rounding the aft edge so it fits into the rounded fillets you just laid in with the wooden spoon. Notch around the pipe that you glued in. You glue in the core by troweling thickened resin onto the back of the core and then press the core onto the 1708 that you laminated earlier. Then you pull the core onto the lamination by clamping or through-bolting though transom and through 2x4s or 4x4s and tighten the bolts up to draw it in tightly to the lamination on the transom. While it is setting up fill any gaps around the core edge and the hull sides and bottom with thickened resin.
You'll be repeating the fillets in all the corners and then rough cutting in the final 3 layers of 1708 with the 12,8 and 4" overlap, to the inside of the transom core. Then you mix resin and repeat the process you used on the first laminations of cloth. After this all hardens, you have a very strong connection of transom core to hull and will have a very strong surface for the motor to push on.
Sorry for the long reply, hope it answers some of your questions.