Does automation really make us safer?

pilots aircraft automation technology
Is technology intended to eliminate human error disregarding the vital role of pilots? Image: Shutterstock

Melbourne Grammar School student William Flintoft explores the implications of our increasing reliance on automation and Artificial Intelligence in his entry, awarded runner-up in the UNSW Bragg Student Prize for Science Writing, under the 2019 theme “Not-so-smart technology”.

“William’s essay starts with a sentence that grabs our attention – and the rest of his essay does not fail to deliver,” said one of the 2019 judges, science writer Bianca Nogrady.

Read William’s full essay below.

2019 UNSW Bragg Student Prize for Science Writing runner-up essay:

Errare humanum est

by William Flintoft (Year 10, Melbourne Grammar School, Vic)

Captain Bhavye Suneja fights for control of his plane in the last few seconds before it crashes into the Java Sea, killing 189 passengers and crew. A later report would attribute the crash of Lion Air Flight 610 to a faulty sensor which resulted in the automated flight control system correcting the plane’s course by putting it into a steep dive. Similar cases, disturbingly identical, are apparent in other aircraft crashes in the past few decades, with automation technology onboard resulting in a tremendous loss of life. These incidents bring into question whether our pursuit of reducing human error is truly something which preserves safety.

An accident similar to that of Lion Air Flight 610 occurred in early 2019 in Ethiopia, with preliminary reports all pointing to an identical cause: the aircraft control system. These accidents prompted the global grounding of the popular 737 MAX aircraft and raised questions about aircraft automation. The Manoeuvring Characteristics Augmentation System (MCAS system) that automates flight controls is believed to be the cause of the accidents in both cases, though such automation of aircraft machinery is intended to eliminate pilot error.

This pursuit of the separation of humans and their decision-making has dire consequences for human lives, and forces us to examine how we develop technologies, what motivates increased mechanisation, and how to escape the challenges they create. In the case of aircraft automation, although using flight automation systems is ostensibly intended to remove the carelessness of human mistakes, it disregards the vital role which pilots have in the technological interconnectedness of flight systems. For example, pilots need to be able to exercise command over flight controls, and their decision-making is often the deciding factor when saving human lives in the instance of an accident. In this way, automation does not so much aid that interconnectedness, it inhibits it, and it brings into question the relationship between humans and the machines we create.

automation industry
“This pursuit of the separation of humans and their decision-making has dire consequences for human lives, and forces us to examine how we develop technologies, what motivates increased mechanisation, and how to escape the challenges they create.” Image: Shutterstock

Similar to the automation of flight systems, mechanisation in the manufacturing sector is raising issues about the role of people in the equation of efficiency. Although proponents of automating manufacturing tout its increased productivity, the reality is that it displaces people from their occupations without the necessary skill set to find employment. Consulting firm McKinsey estimates that as many as 800 million jobs will be at risk by 2030 because of automation. While jobs will certainly be created in servicing these machines, the loss of human labour at the expense of cost efficiency speaks volumes about the manner in which we design technology.

On the manufacturing floor, robotics is being utilised at an unprecedented level, with newer, smarter software decreasing the need for human supervision. With the mechanisation of industry, we now see a move towards ‘hand-crafted’ goods, with consumers craving the humanity in time-honed skills and in the uniqueness of each individual piece. This is what is so astounding about mechanisation: that it drives us to place greater intrinsic value on things with human error, despite our relentless pursuit of the elimination of human mistakes.

Like mechanisation, artificial intelligence (AI) will come to define the world of tomorrow. However, as many critics point out, it poses a grave danger to us in the way we deploy it, and in the manner in which we design it. There is the potential for systems to take command of instruments of war and manage other computer systems at a vast scale. This development is driven in part by our desire to improve upon the limited capacity of our own human minds, yet again pointing to our insatiable hunger for more efficient and advanced alternatives to humans.

Yet AI has been identified by many, including renowned physicist Stephen Hawking, as constituting a genuine threat to our species. Humanity’s dominance is predicated on our intelligence, and there is a danger that AI will upset that order. Self-improving AI systems deconstruct events and decisions into countless constituent components, rendering them debased and neglecting the human aspect of their actions. In this, it is apparent that human mistakes – such as our questioning the morality of our decisions, even when given overwhelming quantitative evidence to the contrary – are necessary, and something which AI simply disregards.

Humanity’s pursuit of ever ‘smarter’ technology to remedy our childish mistakes is fast becoming something which is, paradoxically, a hindrance. Roman statesman Seneca summed it up best in his saying, ‘errare humanum est’, literally, ‘to err is human’. We are forgetting the human side of technology. We neglect the human complexities and errors which make systems so successful, be it the judgement of pilots, the careful handcrafting of workers or our focus on morality instead of data in our decision-making. If we continue to pursue ever ‘smarter’ technologies that render humans no longer a factor in systems, we risk the lives and livelihoods of countless many individuals.

For the 2019 UNSW Bragg Student Prize for Science Writing we asked Australian high school students to enter 800-word essays responding to the 2019 theme of ‘not-so-smart’ technology, identifying and discussing a problem in the world that has yet to be solved by contemporary science and technology.

Read the other winning entries:

STEM Contributor

Author: STEM Contributor

This article was written by a STEM Contributor for Careers with STEM. To learn more, please visit our contact page.


Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.