The prison population in most jurisdictions is escalating. As many prisoners are also parents, more children will inevitably be affected by the experience of having a parent incarcerated. Police and the lower courts are the gatekeepers of the criminal justice system and make urgent and vital decisions about arrest, remand and sentencing which have critical consequences for the children of those arrested, remanded and sentenced. To better understand how these children are responded to by this adult system, this paper draws on data collected from a purposive sample (N = 16) of Victorian magistrates, legal representatives and police, as part of a broader ARC funded study. Findings indicate that the consideration of these children by police and magistrates is largely ad hoc and depends on good will and the exercise of discretion. The balancing of justice issues and the interests of children is also complex and currently under-researched and under-informed. The authors argue that until the agencies dealing with adults incorporate child-focused practices, children, who have a primary carer in prison, will continue to be disadvantaged by a system which considers them only as collateral damage in the exercise of justice.