Common contexts

Common contexts

Devised by Nicolas Malevé

One-liner

Exploring the problem of classification in image datasets

Learning Objectives

• To sensitize the participants to the complexity of applying categories to images.
• To gain a sense of how computer scientists think about “context”.

Keywords

Classification | image | keywords | context

Main image

Equipment/Materials Needed:

1. Interactive whiteboard, digital projector
2. Printed copies of the grids (see main image and samples below)
3. All participants need access to a computer or iPad, or one between two or three with internet access.

Approximate Duration:

40 minutes

Activity:

An image from the COCO dataset

1. Briefly introduce the COCO dataset. For a machine to identify an object, it needs to learn categories of objects. It also needs to have a sense of context. To do this, the scientists have been "gathering images of complex everyday scenes containing common objects in their natural context" (Here we presuppose that the participants already have an idea of what a dataset is)

2. Briefly discuss the problem. Computer scientist created COCO to teach machines that certain objects are more likely to appear together in an image. They define "context" as the co-occurrence of two elements. For instance, when there is a computer mouse, there is a great likelihood that a computer is also represented in the same image. The teacher can add a little more information found on the website https://cocodataset.org/#home. Open question what kinds of objects do you think are likely to appear in a same image ("share a common context")?

A search on the COCO website

3. Using the search engine of the COCO website, search for the combinations of terms that are represented by icons. For instance in the above image, the participants have selected the combination of banana and computer mouse. By groups of two or three students, search for the combination of two terms together in the search engine and observe the results. The participants are invited to start first with combinations that make sense to them (eg. person and umbrella) and then try stranger combinations (ice cream and zebra)

Prompts for discussion could be displayed:

Representation and identity:

• Are these objects familiar to you? Do you think some of them should be included? Do you eat different food products?
• What do you think about these images? Do the kitchens look like yours? In which country are they taken?

Provenance:

• Do you think these images have been taken recently? Where do you think they come from? Where would you take recent images from?

Image within image:

• Often two terms are used in the same image but they are not really depicting objects. For instance we can find a mouse and a giraffe in the same image because the giraffe is represented in a poster hanging next to a computer. What do you think about that? Do you think a real giraffe and a real computer mouse can be easily found together? In which circumstances?

4. Feedback to whole class. Ask pairs to share interesting results, both similarities and differences.

Extension Activity:

5. Propose a new category with its icon and look for images online that fit that category.

6. Compare and discuss the selections. What have you included and what have you excluded?

Examples of search results

Banana + mouse

Banana + mouse

Banana + mouse

Banana + mouse

Banana + mouse

Teddy bear + elephant

Teddy bear + elephant

Teddy bear + giraffe

Teddy-bear + giraffe

Traffic light

Traffic light + toilet

Laptop + traffic light (by the window)

Notes

This activity is best conducted after a general intro to computer vision and datasets.

Download this Activity

PDF version of the instructions