How to discipline managers of cooperative structured financial institutions (co‐ops) is considered a critical issue by the Japanese financial regulatory authorities because co‐ops play a significant role in the domestic banking market, especially for small and medium‐sized enterprises. This paper seeks to clarify whether the effect of the governance‐related variables on firm performance varies across stock and cooperative banks in Japan. We consider regional banks as a proxy of stock banks and Shinkin banks, one of the representative co‐ops, as a proxy of cooperative banks. We use cost and profit efficiency scores obtained from stochastic frontier analysis as performance measures. The results in this paper confirmed that having a large number of board members has negative effects on efficiency measures for both stock and cooperative banks. On the other hand, the presence of outside directors has a significant effect on efficiency measures for cooperative banks, whereas such variables have no significant effect for stock banks. These results suggest that outside directors' discipline is more necessary for cooperative banks than for stock banks, which are under strong pressure from shareholders. For cooperative banks, a high ratio of representative council members, which is the most important decision‐making body for Shinkin banks, has negative effects on efficiency measures. Our findings support the current proposals of the financial regulatory authorities' council to appoint outside directors to the board as a means for strengthening governance of the co‐ops.