Depending on how your municipality cleans and purifies its water at the sewage treatment plant, you may be able to taste all kinds of unpleasantness in the water. If you have well water, you may taste even more unpleasant "flavors" coming from your tap. See if you can taste any of the following in your water, and then find out how you can remove those unpleasant chemicals and/or minerals.
In water treatment plants, one of the primary cleaning agents is chlorine bleach. On days where the plant employees detect higher counts of dangerous bacteria, additional and copious amounts of bleach are added to the water in the treatment vats. The water is retested to make sure the bacteria has been killed or that the chlorine has been reduced to safe levels for human consumption. The water is mixed with more water so that the bleach is not dangerous and then mixed with a neutralizing agent before being sent back out of the facility to your home. If you can taste or smell the bleach faintly, you should consider using a tap purifier attachment to filter the water more before drinking it.
Rust is supposedly safe for human consumption when present in water, but when your water is orange in color and tastes the way a rusted nail smells, you really should not drink it. If this is a regular issue, you may want to have a plumber examine your pipes and see if the pipes need to be replaced. If the pipes are not the problem, it means that the water coming from the ground (i.e. well water) or coming from the city's water supply is picking up rust on its way to and into your home's plumbing. A reverse osmosis filtration system, either for your whole house or for your kitchen/bath taps, can be installed to address the rust.
Sulfur in your water is the result of the groundwater running through a natural gas pocket and/or rock that contains a lot of sulfur in it. The water picks up the sulfur from one of these two sources, and because it is concentrated, it remains in the water as the water is pumped up through the well into your home's plumbing. You can either dig a brand-new well, or you can install a water softener system that purifies the water for your entire home. This type of water softener addresses all of the other problems you would have with the taste of your water as well.