viva

Well, now I realise that the thesis is only part of getting a doctorate, you need to be able to talk about and defend your work orally as well as in written form.

My viva date is the 22nd March. Being 6 months into a post-doc at King's College London, I do not have the time for preparation that I might like, and am having to be efficient with time.

Having watched some excellent videos on viva preparation and the examination, I've realised that I fall into the 'it will be alright on the day, I know the stuff anyway' camp of PhD viva-ee, and need to pick up my game a bit. Reading the thesis through is not enough. Even though I've been editing chapters for paper publications, this is not going to be enough to prepare me for the viva.

What is good is that I've given two talks on the thesis (or bits of it) recently, to various types of audiences. I'm giving another talk at Warwick university on Monday. This is giving me (a) a broad perspective on what I'm doing, and (b) lots of practice at dealing with a diverse range of questions, some of which were unexpected. I'm also collecting questions on my thesis from all sorts of different people, from my supervisors and collaborators to people who have never come across my work before. This is really useful - I need to practice answering these. I'm also practising summarising my thesis, e.g. during runs or gym sessions(!) and am going to have a practice discussion with my supervisor. Lots of practice going on here!

From the videos I've just watched, I've compiled some pointers, and am including those here so I can refer to them again (and so that perhaps they might be useful to others).  Finally, wish me luck!

1. Overall points and general questions

? Does your overall argument make any sense? and does it address your research questions?

? Why is your thesis important?

? What is the general area surrounding your thesis and how does your thesis fit in?

? What is the scope of your thesis? (and what does it not cover?)

? What are the strengths of your thesis? [don't be afraid to push the bits that are good]

? What are the weaknesses and limitations of your thesis? [recognise, acknowledge and overcome- good to show that you have ideas how to deal with them - but don't invite extra corrections!]

For both these Qs:

? Be aware of the strengths and weaknesses

? Be precise

? Be ready and willing to talk about them

? Expect challenging and interesting questions

? What's missing from your thesis? 

? Are there counter-arguments to your arguments? How would you address these?

? Why did you make the research decisions you did? What alternatives did you consider, and why did you dismiss these?

? What interested you in this research area, and drew you to this area of work?

2. For the day itself

Read thesis, mark up a copy (with page tags so you can find things!) and bring it in to the exam. Also make a ~2-page summary of the key themes as a reminder.

Be prepared to summarise your thesis arguments and findings at the start, in a concise way. Practice this! Should be natural and relatively informal (e.g. not reading from a prepared script but describing it in a conversation to somebody interested)

Some bits of the thesis may not make sense, or the examiners won't read as you expect them to (haven't got the same reading background, etc) - I may need to clarify. 

The examiners want to make sure I wrote the thesis and did the work - of course as I know I did, I need to make sure I demonstrate this in a satisfactory way.

In facing questions, don't get defensive - but be brave! (and diplomatic)

Answer questions succinctly and in an informed, knowledgeable manner. I can take my time if necessary, ask for questions to be clarified, and even start an answer again if I've completely mucked it up! 

finished...??

Well, the thesis is now printed out and at the binders, ready to be picked up and submitted this Friday morning. At the back of my head, I know that this is not the last time I will work on it. I want to submit a paper based on the thesis to a Cognitive Computation journal special issue which has a deadline of 1st November, generate papers out of the thesis and return to it again at viva time. And it's not handed in, yet. An academic's work is never done?

But...

A little celebrating will be done on Friday!

For the moment, the thesis in its submitted form is available at

http://dl.dropbox.com/u/6496211/AnnaJordanousThesis.pdf 

It's a large file (23.8Mb) so I'd like to put it up, chapter by chapter if possible, on academia.edu or similar, and probably link to it on this blog too. But that is to be done another time, let's at least allow myself a day or two off of it!

Looking for survey participants - no specialist knowledge required - the more participants, the better!

I'm running an online survey for my PhD research, about computer systems that improvise jazz. The survey gives you some brief information on four different systems and some demos of their music, and asks you to think about how creative the systems are.

I'd really like a spread of opinions on this - from musicians to non-musicians, people who improvise jazz to people who don't, from computer programmers through to people who have the ability to break computers just by looking at them (!) - and I'm after as many participants for the survey as possible... so...

Please could you help me by doing my survey? The survey is at this link:

http://edu.surveygizmo.com/s3/547547/Computer-systems-that-improvise-jazz-How-creative-are-they

So far it's taken people around 15-30 mins to do. People are saying that they've found it interesting and that even if they haven't known about computer music or jazz improvisation, they have been able to take part in the survey.

Sadly I can offer no payment/sweets/etc for doing the survey - all I can offer you is a warm happy glow for helping a PhD student in need, or if that isn't enough, the chance to procrastinate for a bit and take a break, whilst doing something research-related. Also it's fun to see what music the computer systems can come up with and how they do it, if you're interested in this sort of thing? Thank you in advance, I really appreciate people's help!

Results will be published in my PhD thesis (due for completion September 2011) and I may summarise the results briefly in a blog post here too.

A 'marks' sheet for evaluating how creative a computer system is

My research looks at how best to evaluate computational creativity systems - how creative is this computer program?

Computational creativity is when a piece of software acts in a way which would be perceived as creative if seen in a person.

From this research I have produced a sheet which can be used to evaluate and assess how creative a creative system is. It helps you judge the creativity of the computer program on several different aspects and components (see this previous post on identifying building blocks of creativity).

Here is the sheet.

Feel free to go ahead and use it, let me know how you get on. To cite this, please refer to my PhD thesis (title, more details and papers on this at my academic web site).

If you want even more detail, then you only have to wait till my doctoral thesis is ready in September 2011, or else please contact me (contact details on my web site) - I would love your feedback.

Update: Building blocks of creativity

There has been a lot of useful feedback from my previous post reporting on my empirically-derived definition of creativity.

As a result, and after some more work that I have done, I've updated the results.

These factors form 'building blocks' of creativity - they contribute to the overall idea of what creativity is. This could also be thought of as an ontology of creativity - a collection of information relating to the nature of creativity.

                           Creativity is...

To expand on these 14 factors, here is a diagram with a little more detail in each factor. (Click on the diagram to enlarge it)

Depending on what creative domain you are looking at (e.g. art, music, problem solving, proof generation, language use, design etc.) some of these factors will be more important than others.

From the results of a survey on musical improvisation, this is a breakdown of which factors are more important (and less so) in this type of creativity:

The importance of different components of creativity, in the context of creative musical improvisation. This illustrates the results of a survey of 34 people, spanning a variety of musical backgrounds from novice to professional expert. Note that some factors are considered to have some negative influence on creativity as well as a positive influence, e.g. Domain Competence.

The next step in this work is to use the survey data and above results to evaluate and compare a number of music improvisation systems, to explore which are more creative than others and why. Results coming soon...

Today I defined creativity... Here's the definition

Here are the results of some work that I have been doing to capture in words a definition of creativity, as we understand and use the word 'creativity'.


[Ok so the timing in the title of this blog post is perhaps a little misleading... Rome wasn't built in a day and I didn't do this work all in one day... ]

The idea behind this work is that when we talk about what creativity is, certain words keep on cropping up as they are strongly associated with creativity. If we can capture those words, then they will collectively form a definition of creativity.

What I have done is to analyse discussions of creativity, to find what words appear significantly more often in such discussions (compared to discussions on unrelated topics). This gave me a list of 374 words. I've condensed this list by grouping words together that mean similar things and analysing the results, identifying 20 themes in the words.

Here are my results: 20 aspects of creativity. I propose that the combination of these 20 aspects collectively form a definition of creativity.

What do you think?

MY DEFINITION OF CREATIVITY

As an aside - for those who like to see the nuts and bolts of research:
The main word clusters identified during this
work, that assisted my work along the way

2nd International Conference on Computational Creativity (ICCC'11)

The call for papers is out for the 2nd international conference in computational creativity, to be held in Mexico in April 2011.

I'm planning to submit a paper containing my thesis work in a 'nutshell' (well, the work up to December 2010, when the submission deadline is).

Should be pretty good practice for summarising the key points of my thesis - plus if it gets accepted it will be a great way to promote what my thesis is going to be about, and get some feedback before submitting, with the people who are the target audience for my PhD. So far, people I've spoken to in this research community have been quite interested in what I'm doing, so hopefully there will be a good level of interest in my work at this conference.

expertise of programmer vs expertise of the programmer's program

It's not often I directly disagree with Maggie Boden but...
"only an expert in a given domain can write interesting programs modeling that domain"
[Margaret Boden 1994, What is Creativity? in Dimensions of Creativity p. 115]

(my brain is now busily plotting how to write a painting program)

A question of definition

Some questions have been going around my head recently, in the context of what creativity is. These questions have been along this theme: Are the defining characteristics of creativity actually just multiple recastings of the same thing?

• Can a discovery be useful but not interesting? 
• Similarly, can a discovery be interesting but not useful?

This was inspired by Colton et al 2000, which looked at how 'interestingness' was evaluated by mathematical discovery systems. Here are some more developed thoughts: 

[Q. Can a discovery be useful without being interesting? I think NO in this domain because if some previously undiscovered concept or conjecture is useful then it has interest because it can be used. 

Q. How about in other domains? Depends on what interestingness means in those domains. 

Q. How domain-specific is interestingness-and how generalised can it be?

In pure maths something is interesting if it helps you progress, therefore interestingness and utility are tied together this way. 

Q. In other domains, can discoveries be useful without being interesting? yes e.g. if they are a means to an end and if it is not your primary concern - I guess this applies to maths too - most maths conjectures are not interesting to me - unless I can see them being useful to me or in solving a notorious problem.

Q. Can things can be interesting without being useful e.g. Doug Zongker's "Chicken" paper is interesting but not useful except as amusement (so does it have some value here in its humour - which is of course its main purpose? Hmmm I can't think of things which are interesting but not useful in some sort of way...)]

Continuing on this line of thought:

• Is there any difference between things that are surprising and things that are novel [and can I just use novelty to explain both?]

[Surprisingness is linked into novelty - if something is seen before it is less surprising. But there is more to surprisingness than this  - e.g. the result of a process may be surprising not because it is unseen but because it is derived in a different way - so... in a novel way...?]

References

Colton, S. and Bundy, A. and Walsh, T. (2000) On the notion of interestingness in automated mathematical discovery. International Journal of Human-Computer Studies (53) pp.351-375

Zongker, D. (2006) Chicken Chicken Chicken: Chicken Chicken. Annals of Improbable Research (12) pp.16-21

43 Dodgy Statements on Computer Art - Brian Reffin Smith

Some really interesting statements here, whether or not you agree with all of them. From Brian Reffin Smith's blog. My favourite is #41...

http://zombiepataphysics.blogspot.com/2010/03/43-dodgy-statements-on-computer-art.html

Self-assessment of creativity

Interesting post at James C. Kaufmann's blog on our self-awareness of our own creativity:
The American Idol Effect: Why We're Not Too Good at Judging Our Own Creativity

I would interpret 'metacreativity' as being creative about creativity, rather than being aware of the extent of your own creativity. However there are some useful links in here on experiments investigating how students rate their own creativity compared to the 'expert' opinion (in these cases, the teachers). I wonder if there has been any work on people being creative outside of the student/teacher domain?

Creativity and Cognition conference October 2009 - feedback

The conference overall: A real eye-opener for the types of research going on under the banner of 'creativity'. The conference was single-stream, meaning no picking and choosing of what papers to go to see, just one presentation at a time, which you could choose to attend (or not).

There were disappointingly few people there working in computational modelling or creative computer systems, or in the psychological processes behind creativity, and a surprisingly high proportion of people working in design.

Once I got used to the balance of papers, though, I found the conference much more useful - I could allocate intense concentration to the most relevant papers and just sit back and enjoy the other presentations and pick out some bits that were useful to me. Quite often I found that a talk which seemed completely irrelevant to my research had some quite nice general observations that fitted in with my growing ideas about how creativity is more than just producing an end product, with process, producer and the surrounding environment/audience/influences taking an important role too. (In fact one of the graduate symposium papers, by Carly Lassig, gave a really useful reference to this: a paper by Rhodes).

Graduate Symposium: This is the way for PhD students to do conferences! Organised by Celine Latulipe and John Thomas, the symposium was held the day before the main conference and was a closed session, with only the participants, organisers and invited guests present.

For me, the people there to comment (Celine Latulipe and Ernest Edmonds) were very useful to have around as Ernest Edmonds has links with Sussex and I like the way he thinks about creativity, plus Celine Latulipe is from a computer science background and, in conjunction with her grad student, had a very interesting paper on a creativity support tool evaluator.

It was a good mix of people in there, and though some talks were clearly more relevant for some people than others (reflecting the overall mix in the conference) everyone could make comments and have useful discussion. Shame the symposium was on the same day as a workshop by Linda Candy and Zafer Bilda on evaluating creativity, as I would really like to have gone to that, but that was definitely the only minor point.

Specific things to follow up after the conference:

• As I mentioned above, the reference to a Rhodes paper about the '4 Ps' of creativity looked useful (Person, Product, Press, Process) although I haven't been able to source a copy of it yet.
• A paper by Ricardo Sosa, John Gero and Kyle Jennings fitted very closely with an idea I am starting work on, about modelling a creative society using an agent based system.
• Frieder Nake's (excellent) talk on algorithmic art and creativity underlined an opinion which I come across more and more, that "Machines can never be creative". In other words, if a computer can do it, it isn't creative because we can see the processes it uses', hence a definition of creativity shifts with the times as computers do more tasks we would consider creative. I disagree with this (of course! for a PhD on evaluating computational creativity!) but really must acknowledge this debate in my work, although I don't think I want to wade into it too heavily; perhaps more philosophical tools are needed in my academic toolkit before I feel ready to tackle that kind of debate properly.
• The paper presented by Celine Latulipe's student, Erin Carroll, and a conversation with Erin afterwards, led me to look at principal component analysis and factor analysis for clustering words together in semantic categories. I'm not sure yet if this is the way to go for this type of task (and have in fact been advised against it by computational linguists!) but it's good to know about.
• Ben Shaw's talk on Emergence in design was very well presented and gave useful links back to improvisation in creativity (particularly mentioning R. Keith Sawyer's work in this area). Also Ben and I had some conversations which led to him giving me some very useful feedback on work I have done with computational linguistics methods - hopefully he found some useful things in my work too.
• Another useful presentation came from Brian Magerko and various other people at Georgia Tech (surely the most represented institution at an international conference that I have ever seen! 1 in 3 submissions from Georgia Tech, I think) This paper was talking about observing improvisation in people, with a view to replicating it in agent-based modelling - this is very closely linked to some multi-agent improvisation simulations I am starting now. While their findings caused some debate later on (particularly whether we use a stored mental model of the world or not), this paper has given some useful thinking material for me as I approach my own work in multi-agent systems.
• An interesting definition of creativity by Viveka Weiley: Creativity = 1. New, 2. Valuable, 3. 'x' - the key question here is what is the 'x' that we are missing out on if we just consider creativity to be tied into the concepts of novelty and value.
• David Norton's talk on DARCI, a computer artist trained using neural networks, was the closest to mine in terms of graduate symposium talks, and we covered a lot of common ground in our presentations. In particular the debate from Simon Colton's paper came up, on whether something was actually creative if it is perceived as creative. Good to hear about the project and it's going to be interesting to see how it turns out.
• The very last talk was a keynote address by Mihalyi Csikszentmihalyi. I have to admit from the book or two I've read of his, I wasn't really expecting the talk that he gave to be very relevant, more based in social comment and individual case studies (and with big conclusions drawn from limited findings...?) I was very pleasantly surprised - the talk was entertaining and useful, with plenty of relevant academic material including a description of attributes of creative people as representing a continuum which creative people can navigate across very deftly as needed.

So that's the Creativity and Cognition conference for me. Plus I met some really interesting people - hopefully some useful contacts! Didn't talk to everyone I had wanted to talk to, and didn't always manage to maintain concentration throughout the conference - by the end of the conference I was more than ready to come back to normal life, it was quite a long week! But I guess you can't talk to everyone, and do everything.

Creativity and Cognition 2009 conference, 27-30 Oct, Berkeley, CA

Off to California to attend the 2009 Creativity and Cognition conference. I'm presenting my PhD work to date at the graduate symposium on the first day then have the rest of the conference to look forward to.  Will record my thoughts here after the conference.

Conceptual art

In a discussion about how you could place a value on a piece of art, I've been introduced to "conceptual art". This is when the artistic process is of far higher importance than the end-product itself; in fact the end-product becomes almost irrelevant - merely a side-effect of the process being executed.

Sol LeWitt defined Conceptual Art in "Paragraphs on Conceptual Art", Artforum, June 1967, as:

In conceptual art the idea or concept is the most important aspect of the work. When an artist uses a conceptual form of art, it means that all of the planning and decisions are made beforehand and the execution is a perfunctory affair. The idea becomes a machine that makes the art.

If assessing how creative a piece of conceptual art is, solely by evaluating the product, then there are two negative consequences:

1. The primary intentions of the artist are ignored (the artist is more focussed on how the art is made than what the result is).
2. The level of creativity presented will probably be underestimated, especially if the art results in producing something that might seem commonplace outside the context of that art installation.
   

For example Tracey Emin's unmade bed, no matter what your opinion of it as a piece of art, was intended to have greater significance than just to picture a dirty bed.

Another example is Duchamp's Fountain (1917): an exhibit of a urinal, entitled 'Fountain', by Marcel Duchamp. With this piece, Duchamp intended the focus to be on how it was interpreted as the choice for this artwork, rather than the physical object itself.

Duchamp submitted the Fountain to an art exhibition for the Society of Independent Artists, where every submission would be accepted and exhibited. Duchamp's submission sparked a debate with the judging panel (of which Duchamp was himself a member!) as to whether this was in fact a piece of art. Eventually the Fountain was included in the exhibition but hidden from sight and Duchamp resigned from the Society board.

Since then though, the Fountain has been judged the most influential modern art work of all time. Who was right, the 500 art experts who made this judgement or the panel who rejected the Fountain as not being a piece of art?

While I don't intend this post to express that the creative process is far more important than the creative product, for me this is interesting evidence as to why process is as important as product.

There is an interesting discussion on the philosophy of conceptual art at Schellekens, Elisabeth, "Conceptual Art", The Stanford Encyclopedia of Philosophy (Spring 2009 Edition), Edward N. Zalta (ed.), URL =

A ground truth for creativity (or lack of)

My officemate and I had quite an interesting discussion yesterday as to whether there is a 'right' or a 'wrong' answer to give, when you assess how creative something is.

Different people come up with different assessments, based on a whole range of factors such as their expertise in that domain, their knowledge of the process by which creative products are produced, personal opinion and bias according to their tastes, and the amount of time and effort which they give to the assessment.

Part of my work involves a computational system which produces so-called creative behaviour and then assesses how creative that behaviour is, as part of feedback into refining the creative process. (I shall discuss this work in more depth in a future post, when it is further developed.)

How do I test whether the system 'works', i.e. whether its behaviour could be deemed creative and its self-assessment of its own creativity is accurate? There is no ground-truth for creativity; in fact this lack of ground-truth is the central research problem I am addressing! To test its performance against other creativity measurements is to test one theory against another rather than testing the model against real life.

So lets test the model against real life - compare its judgements of creativity to judgements made by people. If there is a large crossover and agreement between machine and people, then the machine is making a good job of approximating human assessment of creativity: the only guide we have to follow at present.

Next problem: what if my system is proven to perform well against some 'right answer', coming up with a satisfactory evaluation of the creativity present in the system - does this mean my system is also 'right'? I would say no, it is not coming up with a universally correct assessment: because no such universally correct assessment exists. We can approximate what a number of people would collectively decide about a system or item's creativity, and aim to match the consensus of opinion. More often than not, however, someone will disagree...

Definitions of creativity: common words used

This word cloud is the result of an analysis of definitions of creativity. It highlights the words used most commonly by people when defining what creativity is, in short dictionary-style definitions.

I took 23 online definitions of creativity (retrieved via a Google search, either in the form of a dictionary entry or as a brief definition of creativity). These definitions were merged together in one text file and fed through the online analysis tool at www.wordle.net, resulting in the word cloud pictured.

Besides the obvious appearance of words such as 'creativity', words which appear prominently include new, originality, ability, imaginative, process, produce, ideas and so on. The results show which words we commonly use to describe what creativity is. This gives us a good idea of what factors to examine further when deciding whether something is creative or not.

This work was based on brief definitions of creativity of one or a few sentences long, maximum one paragraph.

Creativity and Cognition conference 2009, Berkeley, California

Here's my 1-page summary of my research, which I am presenting in California in October. The work I'm presenting is starting to develop again nicely, with some case studies which I'll probably blog about in the next few posts.

Time plan for work for the next 18 months...

This timeplan looks super organised but was actually the result of a long train journey, the back of an envelope, a desire to stop reading papers and a slight panic about whether it was even possible to finish my PhD in three years. Time will tell - but now I have some real motivation to stick to this plan - it shows it may even be possible! Here's hoping I stick to it...

Abstract of my research to date - submitted to the London Hopper Colloquium in May 2009

Evaluating Computational Creativity: Excuse me, Computer, How Creative Are You?

Anna Jordanous

DPhil researcher

Music Informatics Research Centre / Creative Systems Lab

Department of Informatics

University of Sussex

a.k.jordanous@sussex.ac.uk

Creativity is a highly subjective concept; people have an intuitive understanding of what creativity is, but may struggle to capture what the word ‘creative’ means in a formal definition. This poses problems should one want to describe how creative something is, or compare it against another item in terms of the creativity being demonstrated. Such descriptions and comparisons are vitally useful both for reflections on previous creative work, and comparison of progress in creative-led practice. In particular, when looking at examples of computational creative systems, how can one machine-produced artefact be compared against others? Or against human-produced artefacts? To measure creativity I use several related aspects of creativity, such as novelty, usefulness, fluency of ideas etc., which are more tightly defined and therefore easier to evaluate (either by assessing people’s opinions or by using quantifiable tests). My research offers a set of tools to measure creativity from a variety of viewpoints: from statistical tests of how unique a creative piece of work is in a given domain, to social opinions on the significance and usefulness of that piece of work. These tools form a methodology for a standardised evaluation of creativity across many domains, for comparison and reflection.

Research hypothesis

A first attempt at defining the exact hypothesis/hypotheses I will be testing in my work. Any comments welcome!

• Creativity is something which can be measured and evaluated along a continuous and open-ended scale.
  
• Two or more items can be compared against each other on how creative they are,
  
• Crucially, creativity is multi-dimensional, involving several constituent factors or parts, considered in combination.
  
• Creativity can definitionally be broken down into a comprehensive set of factors that are more tightly defined and less subjective than creativity itself.
• We can use this set of factors to measure creativity, by assessing to what extent each factor is present. Combining these results is a sufficient way of measuring how creative an item is and gives an equivalent assessment to that obtained by having humans carry out evaluation of creativity in the same items.
• Hence... One can assess the level of creativity present in an item or system, by measuring the system using a set of criteria which together contribute to a definition of creativity.

This set of factors I keep referring to is a work in progress... in its current raw form it includes (in no particular order):

1. Novelty
2. Generation of things
   
3. Process/Methodology
4. Generating something greater than the sum of its parts
5. Going from known to unknown
6. Unexpected
7. Usefulness
8. Experimentation/divergence
9. Evolution of ideas
10. aesthetics
11. non-linear progress
12. Multi-dimensional
13. grounded in knowledge
14. abstract thinking
15. intuition
16. vision
17. flexibility/adaptability
18. intelligence
19. intention
20. environmental/social constraints
21. self-belief/tenacity
22. promotion (personal and by others)
23. fluency of ideas
24. judgement
25. imagination