The concept of a truly autonomous weapons system—a system which is capable of operating itself, independently from human oversight—sounds more like science fiction than science fact. However, the reality is that weapons development is increasingly moving in this direction. Despite reassurances that humans will always be ‘in the loop’, significant amounts of autonomy have been given to certain weapons systems already. Such weapons present unique regulatory problems, arising not so much from their nature as weapons, but from their replacement of the human role in war and killing. This article considers the implications of increasing weapon autonomy for the humanitarian law principles of distinction and proportionality, and the concept of accountability for breaches of international humanitarian law.