My officemate and I had quite an interesting discussion yesterday as to whether there is a 'right' or a 'wrong' answer to give, when you assess how creative something is.
Different people come up with different assessments, based on a whole range of factors such as their expertise in that domain, their knowledge of the process by which creative products are produced, personal opinion and bias according to their tastes, and the amount of time and effort which they give to the assessment.
Part of my work involves a computational system which produces so-called creative behaviour and then assesses how creative that behaviour is, as part of feedback into refining the creative process. (I shall discuss this work in more depth in a future post, when it is further developed.)
How do I test whether the system 'works', i.e. whether its behaviour could be deemed creative and its self-assessment of its own creativity is accurate? There is no ground-truth for creativity; in fact this lack of ground-truth is the central research problem I am addressing! To test its performance against other creativity measurements is to test one theory against another rather than testing the model against real life.
So lets test the model against real life - compare its judgements of creativity to judgements made by people. If there is a large crossover and agreement between machine and people, then the machine is making a good job of approximating human assessment of creativity: the only guide we have to follow at present.
Next problem: what if my system is proven to perform well against some 'right answer', coming up with a satisfactory evaluation of the creativity present in the system - does this mean my system is also 'right'? I would say no, it is not coming up with a universally correct assessment: because no such universally correct assessment exists. We can approximate what a number of people would collectively decide about a system or item's creativity, and aim to match the consensus of opinion. More often than not, however, someone will disagree...
Subscribe to:
Post Comments (Atom)
Posts
Creativity is about more than a set of end-products; the attributes of the creator, the environment in which the creator works in and the creative process all play their part when we judge on the creativity being displayed.
ReplyDeleteIf I am to ask people to judge the creativity of something then I must consider not only their comments but also their opinion on what they are being asked to do:
- Is this task easy/natural to perform?
- If not, what makes this task difficult/artificial to carry out?
- What further information would be useful in order to feel more confident about the assessment of creativity that is being made?
It is this last question in particular that, I believe, holds some intriguing insights as to what is important for the evaluation of creativity.