No, I guess I didn't - how to I connect to that?
No sure what you mean by this so...
If you turn on the key and the gauge moves (and you stated it did) then it is wired to the ignition, that is it receives "switched" power (the ignition key is the switch). If you have several gauges, they are usually wired as shown below.
Unless you rewired the gauge when you did the new tank install, you should be good but.....
If the gauge is receiving constant power (not switched) because it is wired directly to the battery then it will be a constant drain on the system. That stated, your description of its function doesn't indicate that.
Reviewing how the gauge functions, it receives power when you turn on the ignition (key). The needle moves in relation to the amount of resistance to ground in the circuit. This resistance is governed by the sender mechanism in your tank. You have (I'm assuming here) a reed type (aka WEMA) sender which uses a float to control the variance in resistance. Float at the top creates the least resistance (tank is full) and the gauge needle is indicating a full level. Float at the bottom offers the most resistance (tank empty) and the gauge indicates an empty level.
With that understanding, wired through the switch (ign. key) you will not be drawing power through the gauge with the key in the off position. Did you try tapping the gauge?
Understanding how the gauge works could indicate a malfunction with the gauge itself but before you remove and replace the gauge, again I suggest you check all connections. If you remove all the connections on the back of the gauge, it should read empty. I'm sure others here know more about the makeup of the gauge but I believe that it has two magnets inside. Mag 1 is a constant strength (?) that pulls the needle to empty while the other, mag 2 is an electro magnet that works off the resistance addressed above. The less resistance to the current supplied increases the magnetic strength and pulls the needle away from the other magnet. If the needle is sticking or mag 1 isn't as strong as it once was, the needle may not fully return to empty when you cut power (hence the suggestion to "tap the gauge"). Again, you will not be drawing current at this point if wired as shown above.
How about some pics of the gauge when the key is off and then on.
Good luck.