First, there are a couple terms we need to identify and know:
Lossy compression:
Data compression technique that results in some amount of data being lost from the source file, typically eliminating redundant or unnecessary information in the file to save file size. Also provides greatest compression ratio.
Lossless Compression:
Data compression technique that results in no data being lost, provides less compression than lossy, but maintains greater quality by not limiting information stored in the file.
JPG (also known as JPEG):
Short for Joint Photographic Experts Group, the committee that got together and wrote the the standard for the .jpg file format. JPG is a lossy compression technique made to compress colour and gray scale images that supports 16.7 million colours. JPG is best for complex graphics and photographs; however there is often a loss of sharpness and clarity when used for text or line drawings.
GIF:
Short for Graphic Interchange Format. This is a lossless compression technique that only supports 256 colours. GIF is better than JPG in respects to images that have fewer colours, crisp lines, and black and white images. GIF also supports animation and transparency, but only to a degree of a single shade value (IE: all black can be transparent, or all white, but not both). A major limit to GIF is that the algorithm used for compression is owned by Unisys.
PNG:
Short for Portable Network Graphics. A patent-free alternative to GIF that also has many improvements over GIF compression. Also lossless, it can often gain a 5%-25% more compressed file than GIF, and supports up to 24bit colour. PNGs also allow the use of opacity, which means that there is a varying degree of transparency from one shade to the next rather than GIF’s required sharp edges. PNG does not support animation, and is not supported by all browsers (mainly IE6 and below).

Ok, Now What?
Now that we know the differences, it’s time for the whys. While the GIF is probably one of the more popular formats on the net, it’s quickly being replaced by PNG because of the ability to control opacity. GIF MUST use either 100% opaque or 0% transparent, PNG’s ability to control opacity means that you can hit everything in between, allowing gradients to be used. In January I’m going to be covering some techniques where you will see this used.
So when it comes to our web sites where should we use what? GIF goes well for logos; unless you have a gradient then you use PNG. If we have a lot of small icons, GIF is a great way to go most of the time, if you have thumbnails of images keep them at a JPG though.
So we see the GIF uses being limited, when should we compress with PNG over JPG? A general rule is smaller the image dimensions, the better it will fare as a PNG. PNG works better on vector graphics and anything with a hard line or complex gradient. However, with larger images PNG is horrible on compression. I typically make most all of my repeatable elements PNG as I can make them small, use opacity, and achieve greater compression ratios; while my more detailed, larger images are JPG.
So to give you an idea of the actual compression of each:
GIF(256): 4:1 - 10:1
JPG(high): 10:1 - 20:1
JPG(med): 30:1 - 50:1
JPG(low): 60:1 -100:1
PNG: 5-25% smaller than GIF.

