Answer - It is difficult to answer your question with confidence without more information.
First, I am not sure what you mean by "sour" or "off". It is very unlikely your milk actually goes sour (that is fermented by lactic acid bacteria) as this doesn't occur at refrigeration temperatures. It is more likely the milk goes "off" which means the flavor changes, usually as a result of growth of cold-loving (psychrotrophic) bacteria.
There are two or three potential reasons why the skim might "go off" faster. First, the rate of spoilage depends on how often it is in and out of the refrigerator. If the skim is out more often than the fat-containing milk, it would spoil faster. Second, because of the lack of fat in the skim milk, any small change in flavor is noticed more quickly then in fat containing milk. Finally, it is possible, if all other things are equal, that the free fatty acids (naturally present in the fat containing milk) are acting as natural preservatives in the system.
Note the above answers assume you purchased these products at the same time, and they have the same original date of manufacture (or sell by date) and were manufactured by the same company.
Bob Roberts, Associate Professor of Food Science