1 Introduction

Meetings are a normal part of business and academia. For example,company employees regularly hold meetings for which the time and place isdetermined beforehand. Meetings are an important mechanism for exchangingopinions and establishing social networks. There are three types of meetings:the face-to-face type, the synchronous virtual type like a remote meetingsystem or chat system, and the asynchronous virtual type like a bulletin boardsystem (BBS) or e-mail communication. We focus on face-to-face meetings becausethey allow written documents to be exchanged and discussed, the participantbehaviors and feelings to be observed, and presentation materials to be viewed.

The advantages of face-to-face meetings justify the time and expense of settingthem up. They are not limited in terms of the media used for exchanginginformation (written text, voice, etc.) or in terms of how they can be accessed.For instance, the participants in a face-to-face meeting about a robot and webapplications under development will be more likely to express their opinions ifthey can touch the actual robot or operate the applications than if they canonly view photographs and/or animations. If something is discussed in aface-to-face meeting, the participants are likely to understand the contentmore thoroughly than if they had discussed them in an asynchronous meeting.

There have been several studies on visualizing the structure of a discussion.The graphical issue-based information system (gIBIS) supports theunderstanding, arranging, and retrieving of the remarks made during adiscussion by semantically structuring them and then visualizing the structureduring the course of the discussion. Matsuura described amethod that supports the understanding of the general stream of a discussion bystructuring and visualizing the stream using surface analysis based oncharacters in the minutes.

We previously developed a technique, called "discussion mining", in which content is semi-automatically generated froma discussion. This content can include text information, audio-visual media,and metadata. We have now developed two systems for visualizing this content. A"discussion visualizer" system visualizes the structure of the discussion inthe current meeting, and a "discussion reminder" system enables the contentsgenerated for previous meetings to be retrieved and browsed. These systemsfacilitate understanding of the contents of the current meeting and thuspromote more active discussion.

2 Discussion Mining

There are two targets of discussion mining: one is to generate content from adiscussion by recording and structuring the text information, audio-visualmedia, and metadata presented in a face-to-face meeting, and the other is tosupport subsequent knowledge activities through the use of the generatedcontent. We have been using our discussion mining technique for several yearsand have created several hundred files consisting of content generated frommeetings.

Fugure1: Increase in number of remarks/statements by belonging period.

The systems we have now developed for retrieving and browsing the contents ofprevious meetings and for visualizing the structure of the discussion in thecurrent meeting support the participants in the current meeting and help toactivate the discussion. This enables the participants to ascertain thedivergence and convergence of the current discussion and to more equally andactively participate in the discussion.

Fugure2: Number of agreed and marked remarks/statements by belonging period.

An effective way to enable participants to ascertain the divergence andconvergence of a discussion is to help them visualize the ongoing discussion.The system we have developed for visualizing the structure of the currentdiscussion enables the participants to better understand and order the contentand to contribute more effectively to the discussion. For example, aparticipant might transfer the right to speak to another participant who feelsthey have something important to contribute. If only a few remarks have beenmade about an important topic, the participants might try to discuss it moreactively. On the other hand, if there have already been too many remarks, theparticipants might try to narrow the topic down.

One reason a discussion might stagnate is that the participants have unequalknowledge about the topic. Those participants who have considerable knowledgeabout the topic and/or are familiar with previous discussions about it canbetter understand the current discussion and are more likely to participate (wecall them "high-level participants"). Those who do not are less likely toparticipate because they may lack sufficient knowledge to understand thediscussion (we call them "low-level participants"). Figure shows the number of remarks/statements in a yearby periods that a speaker has belonged to our laboratory. If we regard a personwho belongs for a short period a low-level participants, we can see thatlow-level participants less likely to participate. Correspondingly, theseparticipants make fewer profitable remarks/statements. Figure shows the number of agreedremarks/statements and marked ones, i.e., those that mean some participantsfelt were important. A way of acquiring information about agreed and markedremarks/statements is mentioned in Section 3. We can lower the threshold ofdiscussion and thereby make it easier for low-level participants to participateby presenting information that supplements the discussion.

The two systems we present here help the participants to understand and arrangethe contents of the current discussion by visualizing information about thecurrent and previous discussions. Before describing these two systems, webriefly explain the "discussion recorder" system, which supports ourdiscussion mining technique.

3 Discussion Recorder

There has been much research into the automatic creation of meeting minutes,. Lee developeda method that records the participants' actions and comments using cameras andmicrophones and then produces indexed minutes using auto-recognitiontechnology. Chiu described a system that integratesaudio-visual media, text information, and presentation materials.

Fugure3: Discussion room.

Fugure4: Discussion segment.

With our discussion mining technique, we use a "discussion recorder" systemto document, retrieve, and browse remarks and statements made during a meeting.The recorder also enables each participant to reserve the right to speak. It istargeted at meetings that include a presenter, a secretary, and participantsand in which the presenter presents her or his ideas using slides in adiscussion room. Figure illustrates an example setup forsuch a meeting.

Fugure5: Correction of parent remark/statement associated with current remark/statement.

The information gathered by the discussion recorder is presented on a mainscreen and two sub-screens. Microphones and cameras record the contextinformation as audio-visual content.

The presenter uses a browser-based interface to display and change slides.Information related to the presentation operations, such as the begin and endtimes, is recorded automatically.

Immediately before a participant begins speaking and immediately after she orhe finishes, she or he holds up a "tagging device," which transmits his orher ID and statement type (given below). This enables the recorder to linktheir remarks with the corresponding scenes in the recorded audio-visualcontent. Participants can also use the tagging device to indicate agreement ordisagreement with the statements and remarks of the presenter or otherparticipants, to mark ones for later retrieval. The information on operation iscollected and recorded by the discussion recorder. The recorder can alsoevaluate the remarks and identify the majority opinion.

There are two types of remarks/statements: "start" and "follow." The startones trigger a discussion, and the follow ones contribute to an ongoingdiscussion. We use these types to automatically segment the discussion toenable analysis and efficient browsing of videos. These "discussionsegments" have a tree structure in which the start remark/statement is theroot node and the follow ones are ancestor nodes (Figure).

As mentioned, the discussion recorder enables participants to reserve the rightto speak. A participant can clearly indicate that he or she wants to speaknext. This makes it possible to clarify the order of speaking, which helpsprevent cross-talking. If a reservation is made for a follow remark whensomeone is speaking, the system records the reservation as a remark followingthe current remark and associates the reservation with the current remark. If areservation is made for a follow remark when no one is speaking, it isassociated with the last remark. The participants can also correct parentstatements by using the tagging device (Figure).

The secretary records text information of any statements/remarks using abrowser-based interface that works with the participants' tagging devices. Anyinformation input using the tagging devices is added to the browser-basedinterface as nodes. The secretary selects a node and inputs text manually.

4 Visualization of Discussion

4.1 Discussion Visualizer

The discussion visualizer system visualizes the structure of the currentdiscussion and shows it on the sub-displays. As shown in Figure, the discussion visualizer shows a (1) a meetingview, (2) a slide view, (3) a discussion segment view, and (4) a discussionsegment list. The meeting view (at the far right) shows a preview of theparticipant image selected, a list of the attendees, and the elapsedtime. The slide view (at the far left) shows thumbnails of the slides in order,and the one of the slide currently being displayed is highlighted. The speakercan run the slideshow by selecting the thumbnails in this view using thepointing function of a tagging device.

Fugure6: Discussion visualizer.

Fugure7: Sequences in discussion segment.

The discussion segment view (second from right) shows the text of theremark/statement that triggered the discussion and the text of the parentremark/statement of the current one (if it is a follow-type one) at the top.The discussion segment for the current remark/statement is shown beneath thetext. The participants use this display area to correct the parentremark/statement, as mentioned above.

The discussion segment list (second from left) shows the stream of thediscussion. The nodes representing main topics are indicated by rectangles, andthose representing subtopics are indicated by circles. They are displayed as achain structure. The keywords of multiple discussion segments are displayed onthe left, and the keywords of each main topic or subtopic are displayed on theright. The nodes that often correspond to questions and answers are marked witha "Q." The degree of agreement with each remark, as reflected by input fromthe tagging devices, is represented by the color intensity of the correspondingnode. Icons are displayed next to the nodes containing remarks/statementsmarked by a tagging device. This enables the participants to identify whereimportant remarks/statements were made.

Various types of discussion segments are created by the discussion recorder,e.g., short ones with only comments on the presentation and long ones with manyremarks and statements as a result of hot debate. The long segments may havefollow remarks derived from the topic of the start remark/statement, so weconsider the start remark/statement to be the root node of the discussionsegment and the subtopics to be derived from the root node. We conducted anobservation of the subtopics acquired by the discussion recorder. We found thatthere are subtopics in which a participant and the presenter merely repeatquestions and answers. We think that these subtopics are not important forother participants, so the discussion visualizer displays a "Q" in the nodesassociated with these subtopics.

We extract a list of subtopics, defined , from a discussionsegment using the procedure shown below. We call these subtopics "sequences"and define as a set of a start remark/statement or followremark/statement that has children in the discussion segment. A function called returns the set of remarks/statements that are children ofremark/statement (if has no children, it returns ). A functioncalled returns the remark/statement that is the earliestchild of (if has no children, it returns ). Figure shows an example of sequences in a discussion segment.

The information recorded by the discussion recorder can be visualized in twoways: the structure of the current discussion can be visualized in real timeusing a "discussion visualizer" system, and the information from previousdiscussions can be retrieved and visualized using a "discussion reminder"system.

4.2 Discussion Reminder

A review of the contents of previous discussions can lead to sharedpresuppositions and common understanding of the background for the currentdiscussion, which lowers the threshold of discussion. The low-levelparticipants can thus participate more actively, which can lead to thinkingabout things from a new point of view and figuring out solutions for problemsthat have not been solved due to a lack of technology. Furthermore, a reviewwill help prevent discussions redundant with previous ones. Our system forretrieving and browsing the contents of previous discussions is called"discussion reminder."

Fugure8: Discussion reminder.

Fugure9: Collaboration in participants in discussion reminder.

We focused on two things in our development of discussion reminder: an accuratepresentation of the discussion contents and quick retrieval of the contents soas not to hinder the ongoing discussion. Unclear and/or inadequate sharing ofdiscussion contents can disturb flow of discussion. We therefore included afunction for browsing videos of previous discussions to enable more accurateunderstanding.

However, review of previous discussion contents can interrupt the ongoingdiscussion. It is thus desirable to finish the review in almost no time byusing a method for quickly finding particular audio-visual content. To enableefficient review, we added a function to the discussion reminder interface thatenables the browsed information to be narrowed down such as by matching thediscussion content to a query, matching slides to the discussion content,associating statements with the matched slide, and retrieving contentcooperatively with other participants.

There are two types of reminder tool, a personal one and a collaborative one.In the former case, each participant has a terminal. He or she privatelyretrieves a previous discussion by the terminal and indicates the founddiscussion to the rest of participants. The problem with using a personalreminder tool is that because a participant cannot input appropriate queriesfor themselves, there is a possibility of not being able to find the relevant.On the other hand, a collaborative reminder tool enables to various queriesto be aggregated and previous discussions to be efficiently retrieved. We thusdeveloped the discussion reminder as a collaborative tool.

A participant who identifies a previous discussion that he or she wants toreview tells the secretary which queries to input, and the secretary inputsthem on a special interface. Various types of information, such as presenternames, meeting dates, and keywords, can be used as queries. The retrievalresults are displayed on the main screen, as illustrated in Figure. The participants can then perform various operationson these contents by using the tagging device in this interface.

This interface displays (1) a discussion content list, (2) a slide list, and(3) a discussion segment view. The discussion content list displays the titlesof the discussion contents containing the discussion-matched queries. If aparticipant selects a title using a tagging device, thumbnails of the slidescomprising the selected discussion content are shown at the bottom of the slidelist. Participants can preview the larger slide thumbnail at the top of theslide list. The discussion segment view shows information about the discussionsegments associated with the slide selected in the slide list, e.g., thestructures of discussion segments, the speaker's ID, and remark keywords. Italso enables the full text of a remark to be previewed. Participants can browsethe video corresponding to the start time of the remark selected in thediscussion segment view in the video view displayed on the sub-screens.

Two or more participants can search the discussion contents cooperatively. Anyparticipant can preview the texts of remarks or thumbnails of slides whilebrowsing using the video view. If someone selects an incorrect remark andbrowses the corresponding video, the other participants can indicate thecorrect remark by previewing the remark texts or by changing the slide selectedin the slide list (Figure ).

However, conflicts between competing operations can occur. For instance, ifsomeone selects a remark/statement in the discussion segment view while someonebrowses another one using the video view, the previous browsing will beinterrupted. We thus included a function in discussion reminder for excludingoperations other than preview of remark/statement texts or slide thumbnailsduring playback using the video view until the participant who started theplayback stops it or the playback ends automatically. This function enables thecontents of a discussion to be efficiently reviewed without confusion.

5 Conclusion and Future Work

We have described two systems for visualizing discussion structures andcontents generated by our discussion mining technique, which records andstructures information exchanged in formal meetings and generates content fromthe discussion. The discussion visualizer system visualizes the structure ofthe current discussion. The discussion reminder system enables participants ina meeting to retrieve and browse the contents of previous meetings. Thesesystems support the understanding of contents of current and previous meetingsand thus promote more active discussion.

• Development of a system that enables nonparticipants to efficiently browse discussion contents

• Improvement of the systems by using information gathered by reusing discussion contents

Though we did not mention it in this paper, we need to evaluate these systems.The discussion visualizer enables participants to ascertain the divergence andconvergence of the current discussion by visualizing the structure of thecurrent discussion. Thus, participants will make an efficient discussion,one in which participants break or put off a discussion about a sidestream ofa presentation. The relationship between the length of discussion andpresentation structure such as mainstream and sidestream will show the validityof the discussion visualizer. We will also evaluate the discussion reminder bya questionnaire survey to participants. We will ask whether they retrieved theprevious discussions efficiently and whether using the discussion reminderhelped them to understand the content of discussion. We think that to collectdata for relevant evaluation these systems need to be operated over a longperiod. As a result, the number of remarks/statements by low-level participantswill increase and be closer to the number of remarks/statements by high-levelparticipants.

Our future work will concentrate on the two items listed below: