Haiku Generator that Reads Blogs and Illustrates Them with Sounds and Images

1 Introduction

Since paintings produced by the Aaron system [Cohen, 1995] were shown at the San Francisco Museum of Modern Art in 1979, artificial art has become a widely discussed topic that is not limited to computer enthusiasts. However, the main question remains unanswered - was Aaron, the plotter-equipped art generating program, extending its author's "teachings" and actually creating something, or was it a mere "expert system" with newer capabilities incrementally added to its repertoire? The problem, at least in our opinion, lies in subjective evaluation, which until recently was widely avoided in science and engineering. However, we think that the closer that machines are to humans with their endeavors, the less uncanny uneasiness we will feel with machines that paint [Lindemeier et al., 2013], sing [Fukayama et al., 2010], joke [Dybala et al., 2008], generate stories [y P´erez and Sharples, 2004] or write poetry [Colton et al., 2012]. We believe that, similarly to how the discussions about the meaning of "life" after the discovery of DNA calmed down, the meaning (or rather the breadth of interpretation) of words like "creativity" will also evolve, and one day we will agree that machines can produce art that we enjoy and do not recognize as "artificial".

1.1 Our Approach

In our opinion, one of the factors that make us feel uneasy about artificial art is the unnatural situations in which we encounter it. Normally, artificial art must be specifically sought; robots' paintings are not on the walls around you, computer generated songs are not played on the radio, and nor does your robotic vacuum-cleaner crack an ironic joke when your room is too dirty after a party. To change this situation, we decided to work on programs that can become a lively piece in the so-called Internet of Things that already surrounds, or soon will surround us. Such pieces can be an interactive frame on your wall, TV screen, your car radio or a mirror when you shave or do your make-up. To test our ideas, we created a program that can be changed into a blog module, RSS reader or other software that can follow what you read and enrich the reader's experience or enhance your own blog posts that you write. It can read a blog entry and express impressions about the contents by reading on-spot generated haikus, playing related seasonal sounds and displaying images. In this paper we introduce our system and describe the results of experiments showing that enriching artificial poetry with multimedia effects can provide a significantly better experience, when not performed in excess.

1.2 Real and Artificial Haiku

Seventeen-syllable Japanese miniature poems called haiku have fascinated people from both Eastern and Western cultures as a quintessence of "beauty in simplicity", showing that a human being can pass powerful (and beautiful) associations to another person with only several words that are restricted by strict grammatical (three lines of 5-7-5 syllables) and less limited semantic rules (a poem must contain one seasonal word kigo). There were and still are many attempts (already in the late fifties [Lutz, 1959]) all over the world to create a computer program that can generate poetry of different kinds. Although this is also the case with haiku, we have noticed that such attempts are missing a very important feature of this particular type of poetry - the brief commentary style of Zen-like simple bliss about what we feel in a specific moment of "now" ([Tosa et al., 2007], [Wong et al., 2008], [Tosa et al., 2009], [Netzer et al., 2009]): the deep connection with the situation we have just witnessed, the feeling we have just experienced. We believe that in the case of art, especially when we talk about haiku, it is not the algorithmic sophistication of the generation process, it is more about feeling that the poet understands a situation and that we can mentally agree with what she/he (or maybe it) shares with us. With our system, we want to tackle this problem of emotional unison without overlooking the scientific side of the field and providing a deeper evaluation, which is very often neglected by representatives of the poetry generation subfield of artificial art generation.

2 ProposedMethod

The basic approach in our method is to use word occurrences in the WWW and grammar templates extracted from haiku poems created by Japanese poets. The key role is given to related nouns. The whole process of the method is presented in Figure 1.

2.1 WWWUsage

To achieve the above-mentioned feeling of a "given moment", of a situation provoking emotional recollections (usually nostalgic in the case of haiku, because the genre is more about the pensiveness of recollections than conveying vividly distinctive emotions), we concentrated on blogs, which are nowadays a usual means of writing a diary, sharing views and moments online; however, the method could be used with any kind of text, from news articles to entire books. As we work with Japanese language, we decided to utilize Ameba Blog 1 , which is the biggest Japanese portal for bloggers, and Yahoo! Japan 2 for searching the related words and their occurrences. Software written in Perl accesses the Web with the text browser w3m 3 for both scrapping contents and reading search hits.

1 http://ameba.jp

2 http://yahoo.co.jp

3 http://w3m.sourceforge.net

2.2 Algorithm

A blog site with a title and date information (for season recognition) is used as input. The system checks if nouns are present and whether the whole entry is longer than 50 characters. Nouns are recognized using the standard Japanese morphological analysis tool MeCab 4 .

Choosing a Theme

If two or more neighboring nouns are recognized, they are treated as a one-noun phrase. A haiku theme is chosen from a list of frequent (in the input blog contents) nouns, and the system tests the importance of one of these against the others by searching the web for their co-occurrences. For example, when one of the theme candidates was "Mt. Moiwa", the other top five frequent nouns from an entry were: "autumn leaves", "wind", "clouds", "mountain" and "observation deck". The acquired hit number of candidate w becomes No(w), the number of theme candidate w occurrences within a given entry becomes C(w), and both values are used in the following formula:

This scoring method allows us to avoid the most common Japanese nouns becoming theme words. The next step is to see if the candidates (starting with ones with the highest ThemeScore) are appropriate. The system checks their cooccurrences using a simple co-occurrence probability model (See Figure 2), which is calculated with the following formula:

4 MeCab: Yet another part-of-speech and morphological analyzer. https://code.google.com/p/mecab/

where Retr(N1,N2) is the number of search hits for a query containing both N1 and N2 nouns. This probability is used later to score nouns. The values of the top five nouns become N1 (e.g. "autumn leaves") and N2 is the candidate's (e.g. "Mt. Moiwa") value. After repeating this process for all five nouns and summing up the probabilities, the candidate with the highest value is chosen.

Scoring Nouns for Generation

Using co-occurrence probability as explained above, the system calculates words (nouns and adjectives) that are related to both the chosen theme word w, and the title t of the blog. The following method is used, and the closer the co-occurrence of noun N1 to the title and theme, the stronger relationship they have. If the value approaches 1 and the N1 score exceeds 1, it changes into a reverse-ratio score.

Acquiring Appropriate Adjectives

Our system also acquires an appropriate adjective Adj(N1) for every chosen noun by using 20 web search engine snippets 5 (the same approach to co-occurrences is used; see Figure 2). A snippet is parsed and adjectives are extracted. First, the retrievals are validated using the following method:

The retrievals are then scored with:

5 We have experimented with different numbers and lengths of snippets to ensure that the same pairs are retrieved with the fewest number of searches.

where C(N1,Adj(N1) is the number of snippet occurrences for Adj(N1).

In order to confirm whether the items retrieved as the top adjectives are the most related to the blog, we performed a small-scale experiment that showed that the system's first choice (first rank adjective) were preferred by the majority of six human subjects, who read the blogs and evaluated five adjectives from the system's ranking module (the extracted nouns were randomly reordered to preserve fairness). However, as shown in Table 1, values did not decrease constantly, which suggests that we need to test different ranking methods.

Acquiring Appropriate Seasonal Words

Similarly, our system chooses the most appropriate seasonal words kigo from a kigo-database 6 using the following formula:

where Retr(Nk,w) is the number of hits for a query containing both kigo and the theme word, and Retr(Nk) is simply the number of kigo web occurrences. The kigo-database consists of 2,473 seasonal words divided into four seasonal categories: spring (601), summer (802), autumn (525) and winter (545).

2.3 Haiku Text Generation

To create grammatical templates, we have randomly chosen 312 from 20,000 poems found on the Internet, among which most (about 17,000) were written by modern (postwar) Japanese amateur poets, and the remaining 3,000 are masterpieces carefully selected by specialists. We did not choose only from the masterpieces because they often use archaic grammar, and we did not want to be biased towards any particular trends in order to reach the widest user tastes possible. From the randomly chosen poems we extracted the seven most common grammar patterns for the obligatory three lines of haiku which could deal with different syllabic lengths of theme words. It must be underlined here that in the case of haiku poems, the wording, not the form variations play the most important role, and for that reason we limited the number of templates (although this feature can be easily extended). For example, for four-syllable theme words the pattern was:

6 http://www.haiku-data.jp/

• Theme word + possessive particle "no" (first line)
• Three-syllable kigo + possessive particle "no" + threesyllable noun (second line)
• Five-syllable noun or noun phrase (third line)

When, for example, "Mt. Moiwa" mo-i-wa ya-ma, five syllables) was the theme word, a slightly different template (with adjective) was used and the following haiku was generated:

Mount Moiwa is this the view of yellow gingko leafs?

When a haiku candidate is generated, every line is automatically searched for on the WWW. If the exact match query does not generate any results, the next template is used and another candidate is tested. With this simple step, it is rare to produce lexically or even semantically erroneous lines.

2.4 Adding Voice, Sounds and Images

For reading poems out loud we used a popular program for Japanese speech generation, AquesTalk2 7 , and inserted onesecond pauses between the lines. We also prepared 10 sound effects such as birds, cicadas or chimes, which were associated by searching their titles with nouns using the Yahoo! Japan search engine. For instance, if the word "summer" was used in a haiku and "cicada" had the highest occurrence, the sound of the insect was chosen to illustrate the poem (the user presses a button to play it). For the incorporation of images, the system simply retrieved the first image from a Yahoo! Japan image search when a kigo from the given haiku was queried. In the default setup an image is displayed automatically, while the options for reading out loud and sound effects are hidden under buttons, and the user can decide whether he or she wants to listen to them.

3 Evaluation Experiments and Results

First, we evaluated the haiku itself using the Semantic Differential (SD) [Osgood et al., 1957] scale with Varimax [Kaiser, 1958]. This method was used for evaluating human-created poetry in the past [Luber, 1976] and is still being enhanced to measure poetic values [Vala, 2011]. We preferred this impression-based and aesthetic sensibility-centered approach to ROUGE measures for document summarization performance evaluation, which concentrate on importance, relevance and coherence [Yan et al., 2013] or BLEU-based evaluation using poetry experts [Zhang and Lapata, 2014].

The pairs of adjectives we used are shown together with the evaluation results in Figure 3. The haiku poems generated with the proposed method were shown in random order to the subjects (23 bloggers in their twenties, 13 males and 10 females) together with haiku produced using random words from an input blog, which the subjects chose freely. The proposed method achieved 4.51 points on average on a 7-point scale (1 worst, 7 best) while the baseline (haiku generated using input blog words randomly inserted into templates) was 3.68, and a statistically significant difference was confirmed (p < 0.05 in t-test). Normalized Varimax results are shown in Table 2, and the averages in Table 3.

7 http://www.a-quest.com

3.1 Blog Author vs. Non-Author Evaluation

In the next experimental phase, we employed criteria that are used by Japanese professional critics of traditional poetry. We also needed to verify whether being an author of a blog causes differences in haiku evaluation. This time we used 34 people from different backgrounds and ages, half of whom were writing blogs regularly and half with no experience of posting a blog entry (topics were not limited). The bloggers input their own blog entries, and their evaluation was compared with third-person evaluation by the non-authors. We used a common scale from 0 ("I strongly disagree") to 7 ("I strongly agree") and asked the following questions, where the first three criteria are commonly used in professional haiku rating [Tsuji, 2001]:

• Did you have seasonal associations expressed in the input blog?
• Did you feel a poetic lingering specific to haiku?
• Did you imagine the scene/scenery?
• Was the sound rhythm appropriate?
• Do you think it was a high-quality haiku?
• Did the poem related to the contents of the input blog?
• Would you like to use the system on your blog?

The results are shown in Table 4 for blog authors and in Table 5 for non-authors.

The proposed system outperformed the baseline in all categories; however, not every difference was statistically significant. Blog authors evaluated the baseline system at 2.99 and the proposed system at 4.55, and the non-authors at 4.37 and 5.05 respectively, showing that blog writers are satisfied with our systems output, but they are harsher in their critique when compared to the non-authors. The results may suggest that the proposed method does not fully agree with the emotions the authors want to convey with their blogs, and needs further development.

3.2 EvaluatingMultimedia Effects

As the final stage of our experiments, we conducted a comparison between plain text and multimedia-enriched haiku poems. This time we used five randomly chosen blogs from the Ameba portal and showed the system output to 33 subjects (16-25 years old; 19 males, 14 females) divided into three groups of 11 members for different combinations of media. One evaluation scenario had the output poems displayed simply as plain text, and the second scenario had the addition of haiku read by the previously-mentioned voice synthesis software, with sound effects played and a related image displayed. We set a fixed number of blogs in order to avoid generating new poems when a user enters a different blog as input; therefore, in this experiment the question about relatedness to the blog content is omitted. This decision was also made in order to concentrate on the effectiveness of multimedia and to simplify the new type of experiment. As shown in Table 6, the multimedia-enriched system outperformed the text-only baseline, especially in the professional criteria of "Seasonal Impression" (recollections of the annual seasons) and "Lingering Sensation" (poetic depth).

To determine the effect of combining different types of multimedia, we again divided 33 subjects into three groups and asked them to separately evaluate voice, sound and images. It appears that combining all effects together is probably not the best solution, as different sets are stronger and weaker depending on the criteria (see Table 7). For example, the two above-mentioned professional criteria achieve higher scores only in the case of combining text with sound effects, which may suggest that this pair influences our imagination more than when, for example, the text is read by a synthetic voice.

4 Conclusions and FutureWork

In this paper we introduced our haiku generating system, which can illustrate any blog not only with written poetry, but also with sounds and images. We performed experiments on these various outputs to see how the generated haiku were evaluated by lay people, both blog authors and blog readers. Our proposed system outperformed baseline systems in both a text-only comparison and comparison between text vs. text with multimedia. This dominance was also preserved in different types of evaluation, and was especially visible when subjects were asked about haiku characteristics that have been valued by Japanese people for centuries. We discovered that bloggers like the system output but are more critical than readers, which suggests that the generated haiku were often not conveying associations that the bloggers would like to be expressed. However, higher scores in the third-person evaluation by non-authors indicated that blog readers would be more entertained if a blog had a haiku-generating module with the capability of reading poems, playing sounds and displaying season-related images which, using an actual example of a picture of a camisole (Figure 4) as a symbol of a summer word, can automatically change every time the search engine ranking is updated. As the next step in this research, we plan to make a more interactive version by devising a module with learning capabilities to experiment with crowd-based, half human, half machine-based art collaboration. We believe that our new approach targeted at Internet-savvy young generations of Japanese has the potential to entertain them and enrich their reading experience, while at the same time producing high-quality poetry that can be respected also by their parents and grandparents, who have much higher knowledge and wider experience in both reading classic haiku and creating their own. We also plan to experiment with different combinations of semantic differential evaluation in order to enlarge the still relatively small gap between random baseline poems and artificially created ones, although we realize that human beings treat randomness differently when labeled as "poetry". We believe that our work could spark an interesting multidisciplinary discussion on this topic.

Acknowledgments

We would like to thank Mr. Takuya Emori, our former student who was the main contributor for the presented work but never had a chance to share his research with a wider audience.