Genistein, a protein tyrosine kinase (PTK) inhibitor, regulates ion channel activities. However, the mechanism of action of genistein on large‐conductance calcium‐activated potassium (BKCa) channels is unclear. This study aimed to investigate whether the mechanism of Mg2+‐dependent modulation of BKCa channel activity in vascular smooth muscle cells involved inhibition of phosphorylation by genistein or direct interaction between genistein and BKCa channels. The whole‐cell and inside‐out patch‐clamp techniques were used to measure BKCa currents and the effects of genistein on BKCa channel activities in rat mesenteric arteriolar smooth muscle cells. We found that the effects of genistein on BKCa currents were Mg2+‐dependent. Genistein (50 μM) inhibited BKCa currents if the intracellular free magnesium concentration ([Mg2+]i) was 2 μM or 20 μM, but amplified BKCa currents if [Mg2+]i was 200 μM or 2000 μM. The inhibitory effect of genistein on BKCa currents was reversed by the protein tyrosine phosphatase inhibitor sodium orthovanadate (0.5 mM). Daidzein (50 μM), an inactive analogue of genistein, also amplified BKCa currents, and its amplification was insensitive to orthovanadate. Another PTK inhibitor, tyrphostin 23 (50 μM), reduced the open probability of BKCa channels. This inhibitory effect was weaker at 200 μM [Mg2+]i than at 2 μM [Mg2+]i, and was countered by orthovanadate. Our results suggest that genistein amplifies BKCa currents at a high [Mg2+]i, but inhibits BKCa currents at a low [Mg2+]i. The mechanism of this biphasic effects involves PTK‐independent amplification and [Mg2+]i‐PTK‐dependent inhibition. J. Cell. Physiol. © 2014 Wiley Periodicals, Inc.