Mass nouns seem to be rare in math writing. I have done a little poking around the math journals in JStor and have several observations to make. All are tentative observations based on a small amount of evidence (and thinking).
“Space” as a mass noun
“Space” as a mass noun was common before WWII but is rare now. A search for “in space” (in quotes to make it the phrase that is searched for) gives mostly references to outer space and to very old papers, mostly before 1930. Conjecture: The disappearance of mass nouns in math writing is a consequence of the rise of structural thinking in math.
One recent paper where “space” occurs seemingly as a mass noun, in the title no less, is: F. W. Lawvere, Categories of Space and Quantity, in J. Echeverria et al. eds. The Space of Mathematics: Philosophical, Epistemological and Historical Explorations, DeGruyter, Berlin (1992), 14-30. However, the word “space” appears as a mass noun only once in the body of the paper (according to my hasty scan) and many times as a count noun. Anyway I am not sure it is being used as a count noun in the title. It is paired with “quantity”, which is surely an abstract noun, not a count noun.
Areas of math as mass nouns
Areas of math are commonly used as mass nouns, for example, “Using calculus, we see that the function has one maximum”, or “the result follows by straightforward algebra”. The language of math contains several sublanguages with different uses (symbolic language, rigorous language, “rich” language) and one of them is the metalanguage used for talking about doing math, as those examples surely are.
Mass nouns and plurals
In the paper La Palme Reyes M., Macnamara J. and G. E. Reyes (1999). Count nouns, mass nouns and their transformations: a unified category-theoretic semantics, in Language, logic and concepts: Bradford Book, MIT Press, Cambridge, Ma, 1999, pp 427-452, the authors say that plural nouns are mass nouns, in fact they are the free mass nouns corresponding to count nouns under the adjunction developed in that paper. (The Wikipedia article on mass nouns doesn’t seem to regard plurals of count nouns as mass nouns.) Now plurals are mass nouns with atoms (like “furniture” rather than like “water”). Of course, plurals occur all over the place in math writing. Conjecture: In rigorous math prose the only mass nouns that occur are plurals, or at least are mass nouns with atoms.
I am suspicious of the way Reyes, Macnamara and Reyes smush together mass nouns with atoms (furniture) and mass nouns without atoms (water). (“Atom” means in the lattice of parts. “Some of my furniture” can include a bed and two tables, but not the leg of a table. “Water” is treated in language as if it were infinitely divisible. Of course it really does have atoms in the physical sense.)
These two kinds of mass nouns behave differently in many ways. The most important is that plural nouns can refer to either distributive plurals or collective plurals. (“All groups have identities” is distributive, “the voters were in favor of the proposition” is collective.) I doubt that these different kinds of mass nouns constitute a natural grammatical class.