This paper studies the problem of vision-based adaptive control for robot manipulators under a fixed camera configuration, when the camera intrinsic and extrinsic parameters are uncertain. An image-based controller, which requires the image Jacobian for its implementation, is conducted. The image Jacobian is inversely proportional to the depth factor. Hence it is not linearly parameterizable and the depth factor cannot be adapted with uncertain camera parameters. To cope with the inverse dependence of the image Jacobian on the depth factor, the controller employed the polynomial interpolation to parameterize the image Jacobian matrix linearly. The coefficients of the polynomial are obtained by a trained adaptive-network-based fuzzy inference system. A stability analysis for the proposed method is provided by a full consideration of the non-linear dynamics of the robot manipulator. Simulation results are presented to demonstrate the effectiveness of the proposed approach.