The material removal process of micro electrical discharge machining is based on the instantaneous ultra-high temperature generated by a series of repetitive discharge pulses. Due to the size effects, the polycrystal cannot be considered as continuous and homogeneous material when machining is in micron scale, and the effects of material microstructure should not be neglected. In this article, the thermoelectric characteristics of grain and grain boundary are discussed, and the influence of grain size on the machining performances in micro electrical discharge machining is researched. Two kinds of austenitic stainless steels (AISI 304) which are different in grain size are chosen as the workpieces in experiments. It is verified by both theory models and experimental results that the smaller the grain size, the higher the material removal rate, under the same discharge conditions. Both thermal conductivity and melting point of the grain boundary are lower than those of the grain because of the grain boundary segregation. The effective thermal conductivity and local effective melting point of polycrystalline materials vary with their grain sizes since the grain boundary volume fractions change. As a consequence, the material removal rate of micro electrical discharge machining has direct relationship with grain size of the workpiece.