No.020 Whole-Session Evaluation of Interactive Information Retrieval Systems

Sep 20, 2012 Comments Off on Overview

Overview

Overview of this workshop? (please click)

Potential outcome(s) of the meeting:

Framework(s) for evaluation of system support for searchers over a single search sessions
Simulation of search sessions, or at least criteria for such, and
Definitions/measures of search task successs

To accompolish these targets, we plan the workshop heavily depending on the in-depth discussion among the participants followed by the homeworks by every participants and? small number of “Food for Thought” talks.

Links to the Related Activities:

TTD

Sep 20, 2012 Comments Off on Organizers

Organizers

Nicholas Belkin, Rutgers University, USA
Susan Dumais, Microsoft Research, USA
Noriko Kando, National Institute of Informatics, Japan
Mark Sanderson, RMIT University, Australia

Sep 20, 2012 Comments Off on Participants

Participants

(alphabetical order of family names)

??Leif Azzopardi, University of Glasgow, UK
Paul Bennet, Microsoft Research, USA
Peter Bailey, Microsoft Bing, USA
Corrado Boscarino, CWI, The Netherlands
Nicholas J. Belkin, Rutgers University, USA
Ben Carterette, University of Delaware, USA
Charles Clark, University of Waterloo, Canada
Susan Dumais, Microsoft Research/University of Washington, USA
Norbert Fuhr, University of Duisburg, Germany
Viktors Garkavijs, NII, Japan
Hideo Joho, University of Tsukuba, Japan
Kalervo Järvelin, University of Tampere, Finland
Diane Kelly, University of North Carolina at Chapel Hill, USA
Evangelos Kanoulas, Google, Switzerland
Jaap Kamps, University of Amsterdam, The Netherlands
Noriko Kando, National Institute of Informatics (NII), Japan
Gary Marchionini, University of North Carolina-Chapel Hill, USA
Douglas W. Oard, University of Maryland, USA
Jeremy Pickens, Catalyst Repository Systems, USA
Mark Sanderson, RMIT University, Australia
Tetsuya Sakai, Microsoft Research, China
Arjen P. de Vries, Centrum Wiskunde & Informatica, The Netherlands
Max Wilson, University of Nottingham, UK

Sep 20, 2012 Comments Off on Schedule (Provisional)

Schedule (Provisional)

Table version

Monday 8th? October

15:00 –Check-in? (early check-in may be available by request)

19:00 — 21:00 Opening Reception?? (Room Azalea, 1st floor)

—–

Tuesday 9th? October

7:30– 8:30: Breakfast????????? Restaurant Oak, 1st Floor

8:50–10:30: Plenary :???????? Room 208, 2F in the Research Wing.

How to Access:?? From around the Front Desk there are series of? signs to show the directions to Research Wing. We must access there from? 1st Floor.

Welcome
Introduction of participants
“Food for Thought”
- Intro (Nicholas Belkin, Rutgers University)? [slides]
- Analyzing Web Search Session (Peter Bailey, Microsoft Bing) [slides]
- TREC Session Track (Evangelos Kanoulas) [slides]
- Models of Search Behavior (Diane Kelly) [slides]

(coffee break)

11:00-12:00: Plenary

“Food for Thought”
- TREC Session Track (Evangenlos and Ben) [slides]
- User centered (Diane) [slides]
Review of Homeworks (Sue)
- Discussion
  Guidance for “Breakout A” to find the challenges in WSE

12:00–13:30: Lunch — tables are by breakout groups

13:30–15:00: Break-out A to find the challenges in WSE

(coffee break)

15:30–17:30: Break-out A (continued) to find the challenges in WSE

18:00–19:30: Supper

Free time

—-

Wednesday 10th October

7:30– 8:30: Breakfast

9:00–10:30: Plenary:

Report from the Breakout A
select 3-4 topics for furhter discussion

(coffee break)

11:00-12:00: Break-out B, re-organized into 3-4 topics seleced in the? plenary

12:00-13:00: Lunch with break-outB? group

13:30–20:30 : Excursion to Kamakura ?? ?? itinerary

cf.? Kamakura

—–

Thursday 11th? October

7:30– 8:30: Breakfast

9:00–12:00: Break-out B (continued) (incl. coffee break)

12:00–13:30:Lunch

13:30–15:00: Plenary to report the results of the 2nd break-out

(coffee break)

15:30–17:30: Start writing

18:00–19:30:Supper

free time (can continue writing)

—-

Friday 12th? October

7:30– 8:30: Breakfast

9:00–10:30: Writing

(coffee break)

11:00-12:00: Writing

12:00-12:30: Wrap-up:

12:30-13:30: Lunch

?

Venue:

Seminar rooms in the Research Wing of the Shonan Village Center

Seminar Room 208 ( for plenary)
Seminar Rooms 207 and 209 (for breakout)
Lounge (with sofa small table, and a refrige ) avilable 24 hours
- Seminar rooms are availaable 7:00 — 23:00,? can be extended by request

Jun 27, 2012 Comments Off on Homework

Homework

Identify 2-3 “important papers” and/or “key challenges” in whole-session evaluation of interactive information retrieval?
Write a short paragraph on each paper or challenge you identify.

and/or

Write a short paragraph to describe? your experiences in session-based evaluation.?
If there is any URL about it, please include in the paragraph.

WHEN and WHERE to SUBMIT

Please send them to the organizers (Nick, Sue, Mark and Noriko) by Email
by the end of? * Friday? 5th October *

REPORT OUT the HOMEWORK

We will have a session to review the results of the Homework as part of
“Food for Thought” for further discussion.

Jun 27, 2012 Comments Off on Access to Shonan Village Center

Access to Shonan Village Center

Please consult: Shonan Village Center Access Page

(1) Recommendation :

From Narita Airport, JR’s Raid Train “Airport Narita” with “Green Car (First-class coach)” ticket is recommended. It goes “Zushi” Station (nearest station to Shonan Village Center) without any trasfer.? The timetable is here.
It takes 2.5 hours from Narita Airport to “Zushi” Station. “Zushi” is 30th Stations from Narita Airport.
Please make sure to take “Green car”, otherwise sheets are not designed for longer trips and the passengers may stand-up in front of sheeted passengers, etc.? Green car ticket of “Airport Narita” is cheaper than JR’s Narita Express’s ordinary coach, but takes a little longer (30 min or less)
Sheet reservation is NOT available for Rapid “Airport Narita” train.

From “Zushi” Station, please take a bus No.16 from the bus stop No.1 in front of Zushi Station, or a taxi to the Shonan Village Center . The bus stops in front of the hotel where we will stay, and this is the final bus stop of the bus.
There are less-frequent services (1 or 2 service per hour). It takes 15-30 minutes from the Zushi Station to the Center.

?

(2) Second Recommendation (slightly faster, but need to change train)

Take JR’s “Narita Express (NEX)” from the Narita Airport to either Tokyo, Yokohama, Totsuka, or Ofuna, then change to JR’s “Shonan Line”, “Yokosuka Line”,or? Rapid “Airport Narita” train to “Zushi” Station.
Probably it takes about 2 hours from the Airport to Zushi, but may change depending on the connection.
Sheet reservation is mandatory for NEX. You can reserve sheets when you purchase ticket(s) at the Station of Narita Airport.
The NEX and the Shonan Line (including the Rapid “Airport Narita” train) run along almost the same routes from Tokyo to Ofuna. And NEX is slightly faster than the Shonan Line and generally more comfortable in terms of sheets and the services (including the ladies come and go with wagon of snacks and drinks, etc.), so if you can take the NEX longer (i.e. to Totsuka or Ofuna), it is generally faster. From “Ofuna” to “Zushi”, it takes 11 minutes by “Shonan” or “Yokosuka” Lines. “Zushi” is the 3rd station from “Ofuna”.

(Ofuna – Kita Kamakura (meaning is North Kamakura) – Kamakura – Zushi)

Jun 27, 2012 Comments Off on FAQ & Small Tips