Some excerpts I like from Ironies of Automation by Lisanne Bainbridge:
"The classic aim of automation is to replace human manual control, planning and problem solving by automatic devices and computers. However, as Bibby and colleagues (1975) point out : "even highly automated systems. such as electric power networks, need human beings for supervision, adjustment, maintenance, expansion and improvement. Therefore one can draw the paradoxical conclusion that automated systems still are man-machine systems, for which both technical and human factors are important." This paper suggests that the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator. (...) We know from many 'vigilance' studies (Mackworth. 1950) that it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour. This means that it is humanly impossible to carry out the basic function of monitoring for unlikely abnormalities, which therefore has to be done by an automatic alarm system connected to sound signals. (...) This raises the question of who notices when the alarm system is not working properly. "
Why do I blog this? there is a lot more to draw from this paper but I was interested in these two parts because it raises intriguing problems. Automating something (i.e. delegating a function to an artifact, the rationale of design) is not simple and can foster incredible situations.