Common contexts

Devised by Nicolas Malevé, inspired by Philipp Schmitt's Tunnel Vision

One-liner

Exploring the problem of classification in image datasets

Learning Objectives

To sensitize the participants to the complexity of applying categories to images.
To gain a sense of how computer scientists think about “context”.

Keywords

Classification | image | keywords | context

Main image

Equipment/Materials Needed:

Interactive whiteboard, digital projector
Printed copies of the grids (see main image and samples below)
All participants need access to a computer or iPad, or one between two or three with internet access.

Approximate Duration:

40 minutes

Activity:

1. Briefly introduce the COCO dataset. For a machine to identify an object, it needs to learn categories of objects. It also needs to have a sense of context. To do this, the scientists have been "gathering images of complex everyday scenes containing common objects in their natural context" (Here we presuppose that the participants already have an idea of what a dataset is)

2. Briefly discuss the problem. Computer scientist created COCO to teach machines that certain objects are more likely to appear together in an image. They define "context" as the co-occurrence of two elements. For instance, when there is a computer mouse, there is a great likelihood that a computer is also represented in the same image. The teacher can add a little more information found on the website https://cocodataset.org/#home. Open question what kinds of objects do you think are likely to appear in a same image ("share a common context")?

A search on the COCO website for the keywords banana and mouse

3. Using the search engine of the COCO website, search for the combinations of terms that are represented by icons. For instance in the above image, the participants have selected the combination of banana and computer mouse. By groups of two or three students, search for the combination of two terms together in the search engine and observe the results. The participants are invited to start first with combinations that make sense to them (eg. person and umbrella) and then try stranger combinations (baseball bat and zebra)

Prompts for discussion could be displayed:

Representation and identity:

Are these objects familiar to you? Do you think some of them should be included? Do you eat different food products? Do you have these objects, appliances? Do you visit the same places?
What do you think about these images? Do the kitchens look like yours? In which country are they taken?

Provenance:

Do you think these images have been taken recently? Where do you think they come from? (the have been sourced from Flickr) Where would you take recent images from?

Image within image:

Often two terms are used in the same image but they are not really depicting objects. For instance we can find a mouse and a giraffe in the same image because the giraffe is represented in a poster hanging next to a computer. What do you think about that? Do you think a real giraffe and a real computer mouse can be easily found together? In which circumstances?

Discarded objects:

Many times, we see objects that have lost their use value. They are piles of junk in the street or food waste like banana peels. Then they appear in different places and next to other objects. A toilet can be next to a traffic light. What does it tell you about the nature of an object?

4. Feedback to whole class. Ask pairs to share interesting results, both similarities and differences.

Extension Activity:

5. Propose a new category with its icon and look for images online that fit that category.

6. Compare and discuss the selections. What have you included and what have you excluded?

7. Show Philipp Schmitt's Tunnel Vision and discuss it