The design of space systems is a complex and multidisciplinary process. In this study, two deterministic and nondeterministic approaches are applied to the system design optimization of a spacecraft which is actually a small satellite in low Earth orbit with remote sensing mission. These approaches were then evaluated and compared. Different disciplines such as mission analysis, payload, electrical power supply, mass model, and launch manifest were properly combined for further use. Furthermore, genetic algorithm and sequential quadratic programming were employed as the system-level and local-level optimizers. The main optimization objective of this study is to minimize the resolution of the satellite imaging payload while there are several constraints. A probabilistic analysis was performed to compare the results of the deterministic and nondeterministic approaches. Analysis of the results showed that the deterministic approaches may lead to an unreliable design (because of leaving little or no room for uncertainties), while using the reliability-based multidisciplinary design optimization architecture, all probabilistic constraints were satisfied.