viva

Well, now I realise that the thesis is only part of getting a doctorate, you need to be able to talk about and defend your work orally as well as in written form.

My viva date is the 22nd March. Being 6 months into a post-doc at King's College London, I do not have the time for preparation that I might like, and am having to be efficient with time.

Having watched some excellent videos on viva preparation and the examination, I've realised that I fall into the 'it will be alright on the day, I know the stuff anyway' camp of PhD viva-ee, and need to pick up my game a bit. Reading the thesis through is not enough. Even though I've been editing chapters for paper publications, this is not going to be enough to prepare me for the viva.

What is good is that I've given two talks on the thesis (or bits of it) recently, to various types of audiences. I'm giving another talk at Warwick university on Monday. This is giving me (a) a broad perspective on what I'm doing, and (b) lots of practice at dealing with a diverse range of questions, some of which were unexpected. I'm also collecting questions on my thesis from all sorts of different people, from my supervisors and collaborators to people who have never come across my work before. This is really useful - I need to practice answering these. I'm also practising summarising my thesis, e.g. during runs or gym sessions(!) and am going to have a practice discussion with my supervisor. Lots of practice going on here!

From the videos I've just watched, I've compiled some pointers, and am including those here so I can refer to them again (and so that perhaps they might be useful to others).  Finally, wish me luck!

1. Overall points and general questions

? Does your overall argument make any sense? and does it address your research questions?

? Why is your thesis important?

? What is the general area surrounding your thesis and how does your thesis fit in?

? What is the scope of your thesis? (and what does it not cover?)

? What are the strengths of your thesis? [don't be afraid to push the bits that are good]

? What are the weaknesses and limitations of your thesis? [recognise, acknowledge and overcome- good to show that you have ideas how to deal with them - but don't invite extra corrections!]

For both these Qs:

? Be aware of the strengths and weaknesses

? Be precise

? Be ready and willing to talk about them

? Expect challenging and interesting questions

? What's missing from your thesis? 

? Are there counter-arguments to your arguments? How would you address these?

? Why did you make the research decisions you did? What alternatives did you consider, and why did you dismiss these?

? What interested you in this research area, and drew you to this area of work?

2. For the day itself

Read thesis, mark up a copy (with page tags so you can find things!) and bring it in to the exam. Also make a ~2-page summary of the key themes as a reminder.

Be prepared to summarise your thesis arguments and findings at the start, in a concise way. Practice this! Should be natural and relatively informal (e.g. not reading from a prepared script but describing it in a conversation to somebody interested)

Some bits of the thesis may not make sense, or the examiners won't read as you expect them to (haven't got the same reading background, etc) - I may need to clarify. 

The examiners want to make sure I wrote the thesis and did the work - of course as I know I did, I need to make sure I demonstrate this in a satisfactory way.

In facing questions, don't get defensive - but be brave! (and diplomatic)

Answer questions succinctly and in an informed, knowledgeable manner. I can take my time if necessary, ask for questions to be clarified, and even start an answer again if I've completely mucked it up! 

finished...??

Well, the thesis is now printed out and at the binders, ready to be picked up and submitted this Friday morning. At the back of my head, I know that this is not the last time I will work on it. I want to submit a paper based on the thesis to a Cognitive Computation journal special issue which has a deadline of 1st November, generate papers out of the thesis and return to it again at viva time. And it's not handed in, yet. An academic's work is never done?

But...

A little celebrating will be done on Friday!

For the moment, the thesis in its submitted form is available at

http://dl.dropbox.com/u/6496211/AnnaJordanousThesis.pdf 

It's a large file (23.8Mb) so I'd like to put it up, chapter by chapter if possible, on academia.edu or similar, and probably link to it on this blog too. But that is to be done another time, let's at least allow myself a day or two off of it!

The loneliness of the long-distance writer... or misery loves company?

I will hand in my PhD thesis this September. It is official. I handed in a form today that says as much.

Right now, two months seems like both an incredibly long time to be refining a document and an incredibly short time to finish everything off satisfactorily. Having spent July intensively writing up my thesis work (then getting ill, then getting back to the intensive write-up), I'm starting to produce something that I feel really proud of, slightly sick of and which continually surprises me with where it takes me, even now that all the data is collected and collated.

What has really surprised me during this time is the little community of 'writer-uppers' and 'finishers' that has developed around me.

I'd expected that these few last months of PhD would be lonely, spent mostly in front of a computer screen. That has turned out to be half true: the computer screen and I have become extremely well acquainted.

But... there seems to be so many people around me who are in the same boat, aiming for that September deadline. Even those not doing PhDs seem to be getting into the act, with contracts finishing in September, new jobs starting then, gig calendars starting to look towards October, etc. If I'm stressing about a particular problem in the thesis, or going slightly numb from trying to turn a dry set of numbers into some fascinating facts, then I don't have far to look before I find someone who can emphathise, or someone for a tea break.

So despite the title of this blog post, the tail end of the PhD tale isn't turning out to be like the loneliness of the long distance runner, bravely struggling on alone in the face of all adversity and difficulty. In all honesty it hasn't been that miserable either - in fact on occasion it has even been... almost... enjoyable! (I may have to keep reminding myself of that sometimes in the near future though?)

So, enough procrastination - it's back to the dry sets of numbers then. 2 months and counting till hand-in and the start of the next challenge.

The perils of word counts in LaTeX

I'm currently writing up my thesis. I am writing it in LaTeX, which for the uninitiated means that rather than writing it in Word or similar, I write it in a text file and include lots of formatting commands around the thesis to say `this bit should be emphasised' or `this bit is a subsection', then LaTeX produces a pretty pdf file from my text file.

Like any self-respecting numbers girl, I like to keep a watch on how many words I have written that day, and in total. Although I don't have a word limit to adhere to, hitting a good word count for a day makes me feel like I've done good work, even if a lot of it gets cut later...

For a while now I've been dubious about the word counter I've been using: the inbuilt Statistics word counter in TeXShop, which works using detex | wc -w , or in plain English, it strips the thesis file of all the formatting commands and then counts the number of words left. According to this word counter, I have 61138 words, but I've noticed that this total tends to fluctuate: there have been days where I've written a lot and have ended up with less words than I started with.

Trying some alternatives out:

• texcount *.tex: 60239 in text + 1457 in headers + 2469 in captions = 64165
• ps2ascii thesis.pdf | wc -w: 83415 (but this includes the bibliography which is currently 7686 words and any appendices text) 
• copying and pasting the text into OpenOffice: 75222
• copying and pasting the text into Word: 74664
• copying and pasting the text into TextWrangler (a Mac text editor): 74093

So I've anything between 61000 and 83000 words. TeXShop's statistics is instant but inaccurate. Texcount takes about 30 seconds to process then a quick calculation has to be done. ps2ascii also takes a little while and doesn't separate the thesis text from the bibliography/appendices. Copying and pasting, sadly, looks the most accurate (though now I have no idea how many words I've actually got...)

What I mostly want is a tool that measures progress very quickly, for some motivation - so I'll probably stick with TeXShop's Statistics or use texcount, after all of that!

As my housemate says, however,

it doesn't really matter if its 60 or 80 or 100,000 words, just finish it!

Right then: enough procrastination, back to the writing...
---
PS Must mention the excellent LaTeX tutorials written by Andrew Roberts.

A 'marks' sheet for evaluating how creative a computer system is

My research looks at how best to evaluate computational creativity systems - how creative is this computer program?

Computational creativity is when a piece of software acts in a way which would be perceived as creative if seen in a person.

From this research I have produced a sheet which can be used to evaluate and assess how creative a creative system is. It helps you judge the creativity of the computer program on several different aspects and components (see this previous post on identifying building blocks of creativity).

Here is the sheet.

Feel free to go ahead and use it, let me know how you get on. To cite this, please refer to my PhD thesis (title, more details and papers on this at my academic web site).

If you want even more detail, then you only have to wait till my doctoral thesis is ready in September 2011, or else please contact me (contact details on my web site) - I would love your feedback.

Update: Building blocks of creativity

There has been a lot of useful feedback from my previous post reporting on my empirically-derived definition of creativity.

As a result, and after some more work that I have done, I've updated the results.

These factors form 'building blocks' of creativity - they contribute to the overall idea of what creativity is. This could also be thought of as an ontology of creativity - a collection of information relating to the nature of creativity.

                           Creativity is...

To expand on these 14 factors, here is a diagram with a little more detail in each factor. (Click on the diagram to enlarge it)

Depending on what creative domain you are looking at (e.g. art, music, problem solving, proof generation, language use, design etc.) some of these factors will be more important than others.

From the results of a survey on musical improvisation, this is a breakdown of which factors are more important (and less so) in this type of creativity:

The importance of different components of creativity, in the context of creative musical improvisation. This illustrates the results of a survey of 34 people, spanning a variety of musical backgrounds from novice to professional expert. Note that some factors are considered to have some negative influence on creativity as well as a positive influence, e.g. Domain Competence.

The next step in this work is to use the survey data and above results to evaluate and compare a number of music improvisation systems, to explore which are more creative than others and why. Results coming soon...

Today I defined creativity... Here's the definition

Here are the results of some work that I have been doing to capture in words a definition of creativity, as we understand and use the word 'creativity'.


[Ok so the timing in the title of this blog post is perhaps a little misleading... Rome wasn't built in a day and I didn't do this work all in one day... ]

The idea behind this work is that when we talk about what creativity is, certain words keep on cropping up as they are strongly associated with creativity. If we can capture those words, then they will collectively form a definition of creativity.

What I have done is to analyse discussions of creativity, to find what words appear significantly more often in such discussions (compared to discussions on unrelated topics). This gave me a list of 374 words. I've condensed this list by grouping words together that mean similar things and analysing the results, identifying 20 themes in the words.

Here are my results: 20 aspects of creativity. I propose that the combination of these 20 aspects collectively form a definition of creativity.

What do you think?

MY DEFINITION OF CREATIVITY

As an aside - for those who like to see the nuts and bolts of research:
The main word clusters identified during this
work, that assisted my work along the way

2nd International Conference on Computational Creativity (ICCC'11)

The call for papers is out for the 2nd international conference in computational creativity, to be held in Mexico in April 2011.

I'm planning to submit a paper containing my thesis work in a 'nutshell' (well, the work up to December 2010, when the submission deadline is).

Should be pretty good practice for summarising the key points of my thesis - plus if it gets accepted it will be a great way to promote what my thesis is going to be about, and get some feedback before submitting, with the people who are the target audience for my PhD. So far, people I've spoken to in this research community have been quite interested in what I'm doing, so hopefully there will be a good level of interest in my work at this conference.

The first step on a long road, paved with LaTeX tiles and with a gleaming thesis at the end

Today I wrote the first words of my thesis.

Well...

For a while now I've had a thesis plan and a collection of chapter headings and subheadings - but no content. Now today I've started filling in the content.

What I've written today is very much for a first draft, full of [*** NOTES AND REMINDERS ***] and other aesthetically pleasing annotations. I should imagine that at least half of what I've written gets moved around, edited, or discarded, as I get more used to thesis writing.

But...

The first steps have now been taken on this PhD writing-up journey. After an apprehensive start of not knowing what on earth I was going to write as the first words, I just wrote something. Anything. Then changed it. Lo and behold, I'd started writing up. It feels good!

expertise of programmer vs expertise of the programmer's program

It's not often I directly disagree with Maggie Boden but...
"only an expert in a given domain can write interesting programs modeling that domain"
[Margaret Boden 1994, What is Creativity? in Dimensions of Creativity p. 115]

(my brain is now busily plotting how to write a painting program)

A question of definition

Some questions have been going around my head recently, in the context of what creativity is. These questions have been along this theme: Are the defining characteristics of creativity actually just multiple recastings of the same thing?

• Can a discovery be useful but not interesting? 
• Similarly, can a discovery be interesting but not useful?

This was inspired by Colton et al 2000, which looked at how 'interestingness' was evaluated by mathematical discovery systems. Here are some more developed thoughts: 

[Q. Can a discovery be useful without being interesting? I think NO in this domain because if some previously undiscovered concept or conjecture is useful then it has interest because it can be used. 

Q. How about in other domains? Depends on what interestingness means in those domains. 

Q. How domain-specific is interestingness-and how generalised can it be?

In pure maths something is interesting if it helps you progress, therefore interestingness and utility are tied together this way. 

Q. In other domains, can discoveries be useful without being interesting? yes e.g. if they are a means to an end and if it is not your primary concern - I guess this applies to maths too - most maths conjectures are not interesting to me - unless I can see them being useful to me or in solving a notorious problem.

Q. Can things can be interesting without being useful e.g. Doug Zongker's "Chicken" paper is interesting but not useful except as amusement (so does it have some value here in its humour - which is of course its main purpose? Hmmm I can't think of things which are interesting but not useful in some sort of way...)]

Continuing on this line of thought:

• Is there any difference between things that are surprising and things that are novel [and can I just use novelty to explain both?]

[Surprisingness is linked into novelty - if something is seen before it is less surprising. But there is more to surprisingness than this  - e.g. the result of a process may be surprising not because it is unseen but because it is derived in a different way - so... in a novel way...?]

References

Colton, S. and Bundy, A. and Walsh, T. (2000) On the notion of interestingness in automated mathematical discovery. International Journal of Human-Computer Studies (53) pp.351-375

Zongker, D. (2006) Chicken Chicken Chicken: Chicken Chicken. Annals of Improbable Research (12) pp.16-21

Empirical approaches to Performance: Empirical Musicology II conference

Empirical approaches to Performance: Empirical Musicology II conference (25-26 March 2010, School of Music, University of Leeds, UK)

empirical: "based on, concerned with, or verifiable by observation or experience rather than theory or pure logic : they provided considerable empirical evidence to support their argument." 

musicology: "the study of music as an academic subject, as distinct from training in performance or composition; scholarly research into music."

(Definitions taken from New Oxford American Dictionary 2nd edition © 2005 by Oxford University Press, Inc.)

Whilst musicology can traditionally be highly theoretical, this interdisciplinary conference emphasised an empirical approach, presenting a diverse range of different scientific/practical approaches to the study of music. Focussing on music performance, the conference brought together people from a variety of academic backgrounds to share knowledge and methodologies across disciplines.

The chosen two keynote speakers (Eric Clarke and David Temperley) represented two areas of the spectrum of research covered during the conference. From his standpoint as co-editor of Empirical Musicology: Aims, Methods, Prospects (2004, Oxford University Press) and as an influential musicologist in this field over many years, Eric Clarke gave a historical and critical overview of the use of empirical methods in music research, leading to a project he is currently involved in, the AHRC Research Centre for Musical Performance as Creative Practice (CMPCP). David Temperley brought to his keynote his expertise on probabilistic methods of music analysis and music cognition, discussing how musicians control the flow of musical information during performance.

As an educated guess I believe I was one of very few participants who was not based in a music department (unsurprisingly for a musicology conference!) although in fact, several presenters came from multi-disciplinary research groups. The level of interdisciplinarity demonstrated in the talks did ensure that I didn't feel at all out of place academically, from my standpoint as a music informatician. Many methodologies and tools were being applied outside of their traditional domains to explore a wide range of musical detail, taking advantage of what new technologies have to offer the music researcher. Those that stood out particularly, in my memory at least, were:

• Elaine King and collaborators used a statistical ordination technique borrowed from ecology and educational research (canonical ordination, through the software CANOCO) to cluster together data from participants to extract what students considered their main motivations to prepare for assessed performances. (useful for me as I am looking at how best to cluster large sets of data to extract key themes from the data)
• There was a (beautifully presented) talk from Tal-Chen Rabinowitch on work examining associations between musical interaction in groups of children and their emotional empathic development. To measure the children's emotional empathy, three different measures were used as a battery, of which two came from previous literature and the third was devised for this study. (useful for me as I am looking at how best to measure how creative something is)
• Mark Doffman's presentation on jazz musicians' non-verbal communication focused specifically on how different groups of musicians negotiate how to end an improvisation. His research analysed video footage to examine the communicative behaviour of different types of jazz musicians, using this analysis to examine the musical co-ordination that was happening - (I really identified with this, having more than once been in the position of jamming with other musicians, playing a piece, and having no idea how we were going to make the piece end!)

Organised by Luke Windsor, Karen Burland and Elaine King (on behalf of SEMPRE, a society promoting reserach into music psychology and music education), this conference attracted international participants as well as a large proportion of UK-based researchers. The presentations were on the whole of a good academic standard, with excellent keynote presentations and such a variety of research presented that it was easy to find an angle of interest in most talks. It was a shame that the review process was relatively light, with no critical feedback on abstracts sent back to authors, and perhaps an abstract or two appearing in the proceedings that could have done with a little more editing than they received (although as I myself revised my abstract and title a week before the conference, I am in no position to criticise here..!) This is however only a small negative reflection on what was otherwise an excellently-organised, friendly, academically useful and thought-provoking conference.

To conclude - here is my presentation at this conference, looking at how we can empirically capture what it means to be creative as a musical improviser: 

Defining Creativity in Music Improvisation (presentation slides)

How is creativity manifested in improvisation? We have an intuitive understanding of the concept of creativity that we can use introspectively to suggest answers to these questions, both in theory and during performance. If, though, we want to program a computer to generate music in a creative way, the computer does not understand what creativity is. We cannot ask the computer to behave creatively unless we also give some definition of what such behaviour entails. So the problem becomes: how to define what musical creativity is to a computer.

This work uses empirical methods borrowed from linguistics to capture the words which we strongly associate with creativity. An analysis of the language used in dictionary definitions and academic papers on creativity, as compared to everyday language use, has produced a list of words which we commonly use to discuss creativity, e.g. innovation, openness, divergent. After conducting a survey on how these words can be applied in the context of music improvisation, I empirically derive key attributes of creativity in this musical domain which can be used to guide an artificially intelligent musical system towards generating creative musical behaviour.

Creativity and Cognition conference October 2009 - feedback

The conference overall: A real eye-opener for the types of research going on under the banner of 'creativity'. The conference was single-stream, meaning no picking and choosing of what papers to go to see, just one presentation at a time, which you could choose to attend (or not).

There were disappointingly few people there working in computational modelling or creative computer systems, or in the psychological processes behind creativity, and a surprisingly high proportion of people working in design.

Once I got used to the balance of papers, though, I found the conference much more useful - I could allocate intense concentration to the most relevant papers and just sit back and enjoy the other presentations and pick out some bits that were useful to me. Quite often I found that a talk which seemed completely irrelevant to my research had some quite nice general observations that fitted in with my growing ideas about how creativity is more than just producing an end product, with process, producer and the surrounding environment/audience/influences taking an important role too. (In fact one of the graduate symposium papers, by Carly Lassig, gave a really useful reference to this: a paper by Rhodes).

Graduate Symposium: This is the way for PhD students to do conferences! Organised by Celine Latulipe and John Thomas, the symposium was held the day before the main conference and was a closed session, with only the participants, organisers and invited guests present.

For me, the people there to comment (Celine Latulipe and Ernest Edmonds) were very useful to have around as Ernest Edmonds has links with Sussex and I like the way he thinks about creativity, plus Celine Latulipe is from a computer science background and, in conjunction with her grad student, had a very interesting paper on a creativity support tool evaluator.

It was a good mix of people in there, and though some talks were clearly more relevant for some people than others (reflecting the overall mix in the conference) everyone could make comments and have useful discussion. Shame the symposium was on the same day as a workshop by Linda Candy and Zafer Bilda on evaluating creativity, as I would really like to have gone to that, but that was definitely the only minor point.

Specific things to follow up after the conference:

• As I mentioned above, the reference to a Rhodes paper about the '4 Ps' of creativity looked useful (Person, Product, Press, Process) although I haven't been able to source a copy of it yet.
• A paper by Ricardo Sosa, John Gero and Kyle Jennings fitted very closely with an idea I am starting work on, about modelling a creative society using an agent based system.
• Frieder Nake's (excellent) talk on algorithmic art and creativity underlined an opinion which I come across more and more, that "Machines can never be creative". In other words, if a computer can do it, it isn't creative because we can see the processes it uses', hence a definition of creativity shifts with the times as computers do more tasks we would consider creative. I disagree with this (of course! for a PhD on evaluating computational creativity!) but really must acknowledge this debate in my work, although I don't think I want to wade into it too heavily; perhaps more philosophical tools are needed in my academic toolkit before I feel ready to tackle that kind of debate properly.
• The paper presented by Celine Latulipe's student, Erin Carroll, and a conversation with Erin afterwards, led me to look at principal component analysis and factor analysis for clustering words together in semantic categories. I'm not sure yet if this is the way to go for this type of task (and have in fact been advised against it by computational linguists!) but it's good to know about.
• Ben Shaw's talk on Emergence in design was very well presented and gave useful links back to improvisation in creativity (particularly mentioning R. Keith Sawyer's work in this area). Also Ben and I had some conversations which led to him giving me some very useful feedback on work I have done with computational linguistics methods - hopefully he found some useful things in my work too.
• Another useful presentation came from Brian Magerko and various other people at Georgia Tech (surely the most represented institution at an international conference that I have ever seen! 1 in 3 submissions from Georgia Tech, I think) This paper was talking about observing improvisation in people, with a view to replicating it in agent-based modelling - this is very closely linked to some multi-agent improvisation simulations I am starting now. While their findings caused some debate later on (particularly whether we use a stored mental model of the world or not), this paper has given some useful thinking material for me as I approach my own work in multi-agent systems.
• An interesting definition of creativity by Viveka Weiley: Creativity = 1. New, 2. Valuable, 3. 'x' - the key question here is what is the 'x' that we are missing out on if we just consider creativity to be tied into the concepts of novelty and value.
• David Norton's talk on DARCI, a computer artist trained using neural networks, was the closest to mine in terms of graduate symposium talks, and we covered a lot of common ground in our presentations. In particular the debate from Simon Colton's paper came up, on whether something was actually creative if it is perceived as creative. Good to hear about the project and it's going to be interesting to see how it turns out.
• The very last talk was a keynote address by Mihalyi Csikszentmihalyi. I have to admit from the book or two I've read of his, I wasn't really expecting the talk that he gave to be very relevant, more based in social comment and individual case studies (and with big conclusions drawn from limited findings...?) I was very pleasantly surprised - the talk was entertaining and useful, with plenty of relevant academic material including a description of attributes of creative people as representing a continuum which creative people can navigate across very deftly as needed.

So that's the Creativity and Cognition conference for me. Plus I met some really interesting people - hopefully some useful contacts! Didn't talk to everyone I had wanted to talk to, and didn't always manage to maintain concentration throughout the conference - by the end of the conference I was more than ready to come back to normal life, it was quite a long week! But I guess you can't talk to everyone, and do everything.

Definitions of creativity: common words used

This word cloud is the result of an analysis of definitions of creativity. It highlights the words used most commonly by people when defining what creativity is, in short dictionary-style definitions.

I took 23 online definitions of creativity (retrieved via a Google search, either in the form of a dictionary entry or as a brief definition of creativity). These definitions were merged together in one text file and fed through the online analysis tool at www.wordle.net, resulting in the word cloud pictured.

Besides the obvious appearance of words such as 'creativity', words which appear prominently include new, originality, ability, imaginative, process, produce, ideas and so on. The results show which words we commonly use to describe what creativity is. This gives us a good idea of what factors to examine further when deciding whether something is creative or not.

This work was based on brief definitions of creativity of one or a few sentences long, maximum one paragraph.

Creativity and Cognition conference 2009, Berkeley, California

Here's my 1-page summary of my research, which I am presenting in California in October. The work I'm presenting is starting to develop again nicely, with some case studies which I'll probably blog about in the next few posts.

back on track...

Its been a strange month or so, trying to reassess where I am and whats happening with my phd.

In good news, my submission to the graduate symposium for the Creativity and Cognition Conference got accepted, along with some fairly substantial funding to go to California in October.

I'm rescaling what I am doing slightly, examining a particular domain as a test case for assessing creativity, then using this practical work to consider creativity more widely (although this consideration may not be practically realised in this PhD work).

One of my major problems with the assessment criteria suggested by Graeme Ritchie (2007, 2001) is that while it appears to be a methodology for assessing creativity, in practice it is too unspecified to produce meaningful results. As Ritchie is proposing a framework for anything creative, by definition he has to keep the framework generic and cannot define particular parts of the framework (such as: various measurements of the level of novelty in this program's output must be above the threshold of {theta} in order to satisfy certain criteria - so what is {theta}?)

My work suffered from the same criticism: by being very generalisable, it ended up saying almost nothing? It needed more surrounding context to ground it.

So now I'm concentrating on musical creativity again, specifically in jazz improvisation as its an area that interests me within music and there are a number of jazz generation systems of interest, which I am looking up:

• Philip Johnson Laird's work
• GenJam by Al Biles
• Paul Hodgson's various jazz generation systems (described in his 2006 DPhil thesis)
  
• Jeff Pressing
• etc: I'm sure I'm missing lots out...
  

At the same time, I'm going to continue thinking about what is important for something to be considered creative.

Rather than looking for an over-arching and complete classification system for creativity though, I am looking for the important themes that are incorporated in creativity. With a few different practical projects on how to find these themes, added to my own intuitions on what is important, I'll identify a small set, test that set to see how it matches to human assessments of creativity, then re-juggle the set as necessary.

Graeme Ritchie's assessment methodology will be a starting point for me, although I think that what I end up with will be quite different from his suggestions.

I have to bear in mind that what turns out to be important for judging jazz generation systems may well be less important for other domains of creativity (in fact this is almost certainly going to be the case). But I shall cross that bridge when I come to it, rather than worrying about it too much in advance.

References
G. Ritchie. Assessing creativity. In Proceedings of AISB Symposium on AI and Creativity in Arts and Science, 2001.
G. Ritchie. Some empirical criteria for attributing creativity to a computer program. Minds and Machines, 17:67–99, 2007.

2nd year review

Well...

Having been on the receiving end of almost continuous criticism of my work, in my annual review yesterday, I am now at a bit of a loss as to what to do next.

Although I was expecting a bit of a grilling, I hadn't realised quite how much my proposals would get attacked. Although some of the comments were based upon my having not explained certain things properly (for example stressing that the factors I break creativity down into must be more clearly defined than creativity itself, otherwise how do you measure those factors and what benefit is there for that factor being included?), other comments were far more fundamental, questioning my entire approach.

I've gone from having a very clear idea of what I'm doing and why, and a real focus and motivation, to feeling quite lost again. Now I want to arrange a meeting with my supervisors sooner rather than later, as at this stage I wanted to be really getting on with practical work rather than still questioning what exactly I'm doing.

I guess watch this space?

Journals

Went to a fairly non-eventful research training session today, on scientific writing. Beyond this quite useful wikiversity link on how to write in a scientific way, the only real benefit I got from this (apart from a break from marking!) was to think about what journals I want to be aiming for, for publication.

As my research focus has shifted away from music and towards creativity and evaluation this year, many of the journals I have been used to looking at have become less relevant. So now I've set up some content alerts for some new journals and need to get used to looking at different journals (ideally I want to set aside some time for this each week to browse some journal content).

Here's a list of the journals that are quite relevant for me (along with some journals on music/AI that I prob will still find interesting but not necessarily useful for my current work):

• Creativity Research Journal (impact factor 0.57)
• Lecture notes in AI
• Lecture notes in CS (probably not so relevant anymore really)
  
• Leonardo (and related journals)
  
• Digital Creativity
  
• Cognitive Science (impact factor 2.179)
  
• Topics in Cognitive Science (impact factor 9.389)
  
• Cognitive Science: A multidisciplinary journal
  
• Minds and Machines
  
• Journal of Creative Behaviour (0.429) (we don't get it)
  
• Creativity
  
• Creative Review (not peer reviewed, doesn't look that relevant)
  
• Psychology of aesthetics, creativity and the arts
  
• Evaluation
  
• Evaluation review
  
• Research evaluation
  
  
• AISB quarterly (not really a journal..!)

Music-related journals:

• Journal of New Music Research
  
• Computer Music Journal
  
• Psychology of Music
  
• Music Perception
  
• Musicae Scientae
  
• Journal of interdisciplinary music studies
  
• Journal of music and meaning
• Contemporary music review
  

Measuring Consciousness (talk by Anil Seth at Sussex)

Yesterday Anil Seth, one of the researchers at Sussex University, gave a seminar to the COGS research group on: Measuring Consciousness - from behaviour to neurophysiology. An abstract and references can be found at http://www.sussex.ac.uk/cogs/seminars

This idea of measuring something which is not so amenable to measurement is very close to my work in measuring creativity, so this was pretty useful for me. 

In summary

• Anil is advocating that we measure consciousness by combining several measures of properties of consciousness rather than by trying to find one catch-all measure of consciousness
• He is working by examining tests that measure some property of consciousness then intends to combine the results for a more general measure of consciousness
• The measures of consciousness/properties of consciousness are taken from a variety of backgrounds, not just behavioural and neurophysiological measures but also originating from complexity theory and even economics (Granger causality)
• This approach is very similar to mine, the main difference that I can see is that Anil is working more in a 'bottom-up' way, integrating tests together and seeing how the tests match intuition, refining continuously, rather than a more top-down approach of determining beforehand a set of properties of consciousness then finding suitable tests for each property. His is a more immediately hands-on approach (and is I think the approach my supervisor favours for my work) although I'm not sure how he is avoiding the situation where some vital factor of consciousness may be overlooked just because few or no measures exist for it at present

For more detail... read on!

Anil started by justifying the need to make measurements of consciousness as being vital for a scientific study of consciousness: as a proof that consciousness is actually present and to what degree the thing being studied is conscious (as opposed to a discrete yes/no answer to whether it is conscious). This corresponds very closely to my motivation for measuring creativity.

The idea of consciousness existing at different levels - from primary consciousness e.g. being aware of what you see in front of you, to higher order consciousness e.g. being aware of being aware of seeing something in front of you - was quite intriguing. Are there different levels of creativity? Meta-creativity, being creative about creativity? (Is this what I'm doing, being creative about creativity?) It seems like a fascinating question but perhaps not one for my current work, it could be more distracting than practically useful. Anil didn't go into much detail on this either. 

To actually measure consciousness, Anil discussed both behavioural measures and brain-based measures, saying that to measure consciousness you can't just measure one thing, you have to combine several different measurements of different properties. The aspects you measure have to be both differentiable (i.e. you can make divisions between them, and treat them as different things) and also integratable (i.e. you can combine them all together in a reasonable way, they don't obstruct each other). They also have to be measurable. I asked him afterwards if his intention was to use a combination of behavioural and neurophysiological measurements for measuring consciousness, as his talk focussed more on the details of individual measurements rather than the mechanics of combining them. From his answer, I think that this is the intention but in practice the individual measurements are still being refined. 

This approach is very close to my own approach in measuring creativity, the idea of measuring several properties and factors of consciousness/creativity which combine for a more general measurement of that concept. Both work sees a multi-disciplinary input, combining measures from different research areas. Where Anil differs is in his approach to developing the measurement methodology. Rather than determine what properties need to be measured first, then find tests to match them, he finds tests for consciousness, then acknowledges that they only measure part of consciousness, or only fit with certain theories about consciousness, without being useful for other theories.

It seems Anil is combining together all tests that he finds to be useful (by seeing whether they produce a measurement of consciousness that matches what is expected), without worrying about the wider picture of whether this is getting an accurate snapshot of all the properties associated with consciousness. There is an ongoing reflection on how the tests are used, leading to refinement of the tests. This is especially the case when tests produce different measurements of creativity. Studying the reasons why they diverge can give further insight into consciousness, making the tests useful beyond the actual measurements they provide.

Hmm.....

I wonder if this approach will lead to a slight bias in measuring consciousness? The properties that are more amenable to testing will, by the nature of research, probably be better provided for in terms of tests available, whereas those properties that are slightly trickier to test (but nonetheless still equally valid in defining what consciousness is composed of) may not be so well catered for in terms of tests available? This is the scenario I'm trying to avoid with the approach I am taking.

But on the other hand, consciousness (and creativity) is not something that can be reduced to a mathematical formula e.g. 4*complexity + (0.5*awareness) etc etc - so is there any point in seeing which factors contribute more than others? Or is it all too subjective and potentially leading me to over-define creativity? As I'm coming across in a few of the books on creativity I'm reading at the moment, once you make a tight and fast definition of creativity, you run the risk of not measuring creativity itself anymore, but just a subset of creativity. In his talk, Anil was very careful to suggest only a working definition of consciousness, not wanting to spend time discussing the finer points of this.

I know my supervisor is more in favour of my taking a similar approach to Anil's, slowly building up a battery of tests and discarding those which don't seem useful. So far I have not been sure that this bottom-up approach is as suitable as my top-down approach, despite the extra preparatory work it entails in determining how to define creativity via such criteria. But perhaps I can attack the work from both directions, and see which pays off? (or maybe make the two approaches meet in the middle?)

I'll just briefly summarise the content of the rest of Anil's talk, the parts which could be useful for me as I look into what tests I could use to measure properties of creativity.

Measurements of consciousness through behavioural measures:

• Objective: getting participants to make accurate choices under forced decision making conditions
• Strategic control: examining participants' ability to use or to not use knowledge according to instructions (again looking at choices made by participant but now under different motivations for the participant)
• Subject measures: do participants know what they know? (and can they tell us?)
• Post-decision wagering (participants place bets which reflect their confidence in answers given during experiments - Persaud et al 2007, Nature Neuroscience) and other recent measures (e.g. Shields, Ruffman) like allocating confidence ratings to responses in an experimental situation(Anil described work which showed that post-decision wagering was really equivalent to confidence ratings)

Here different measures fit in with different interpretations of or theories about consciousness - the measures and the theories/interpretations are interdependent.

Measurements of consciousness using brain-based measures:

• Various methods of capturing brain activity, e.g. EEG activity, ERP
• While subjects are awake, their brain activity indicates more irregular activity than when they are asleep (and lower frequence EEG recordings? Berger 1929) - indicating that consciousness is only present when we are awake? Or present to a greater degree when we are awake? This doesn't fit in with the work by Tononi that Anil discussed, which treats consciousness as being a capacity for conscious information transfer/activity rather than the actual transfer/activity - Tononi may probably argue that we are just as conscious when we are asleep as when we are awake, its not something that can be switched on or off as we wake up or fall asleep. The capacity for us being conscious remains the same regardless of our actual acknowledgement of being conscious)
• Dynamical complexity as a key indicator of the presence of consciousness (complex behaviour and information exchange in a dynamic system i.e. a system which can change) (Q. So is there a threshold value of complexity, so that if a system's complexity is above this value than it can be deemed to be (to some degree) conscious?...)

Measures of dynamical complexity

• Neural complexity (all the possible combinations of dividing a collection of neurons into 2 subsets, and how much information can be retrieved from these subsets)
• Information integration (the capacity for a system to integrate information dynamically, as opposed to the actual activity - Tononi 2004)
• Causal density (Seth 2005, 2008) Based on Granger causality: the correlation between two variables (in one direction: seeing if activity in one variable helps predict activity in another variable)

Why use different measures in parallel? to capture subtly different aspects and to override small deficiencies with individual measures (and to avoid overemphasising one measure at the expense of others - which measure do you choose to trust). Gaillard et al 2009 is a good example of putting together multiple measures in parallel, agreeing that there is no single measurement for concsiousness but several potential measurements.

How to decide what to measure and what measurements to use? (especially if different measures are getting different results). There are some boundaries of whether something is conscious to some degree, although these aren't so discrete (conscious or not conscious). Intuition helps, and if two measures give different results, this is good for refining the interaction of measures (although surely good for criticising the relevance of the measures as well? Although Anil didn't really discuss that). Implementation details will also play a part in choosing measures: what is appropriate and reasonable to use in a given scenario.

Useful references:

• Behavioural measures of consciousness: Seth 2008 (Consc Cog), Seth et al 2008 (Trends Cog Sci), Persaud et al 2007 (Nature Neuroscience) (post decision wagering)
• Brain based measures of consciousness: Berger 1929
• Measures of complexity: Tononi 2004 (BMC Neuroscience), Seth 2005 (Network Comp Neur Sys), Seth 2008 (Cogn Neurodynamics)
• Structural properties of consciousness: Seth 2009 (Cog Computation), Seth & Clowes 2008 (AI in medicine)
• Using different measures in parallel: Integrating multiple measures in parallel Gaillard et al 2009 (PLOS Biol)

Time plan for work for the next 18 months...

This timeplan looks super organised but was actually the result of a long train journey, the back of an envelope, a desire to stop reading papers and a slight panic about whether it was even possible to finish my PhD in three years. Time will tell - but now I have some real motivation to stick to this plan - it shows it may even be possible! Here's hoping I stick to it...