The new programme will be a kind of video-Google: a search command will enable you to find the right image or video. To do this, the computer programme needs to be able to distinguish a vast number of concepts. A football scene is relatively easy to identify, but how can you explain something more abstract to a computer, for example ‘a world leader’, ‘two people shaking hands’ or, even more extreme, ‘love’ or ‘romance’? Gevers explains how this works. ‘We need to train the computer-software. If we want the programme to recognise an owl, we show it lots of pictures containing owls. The programme itself then looks for characteristics specific to owls.’

The programme will make use of various information sources coupled to an image or video-file on the Internet. ‘We don’t just look at the image itself, but also check whether the concept looked for can be found in text around it. Perhaps the video is undertitled, or there may be other text in the images. We also use e.g. speech recognition: if we know what’s being said in the movie, we can also search for this spoken text. The soundtrack also tells a lot about the videos. And of course we also look for visual information: whether something is moving, what colours are involved and what the transitions between colours are like.

Gevers says that Google is also working on a video retrieval system. ‘They only plan to search for text accompanying the video. However, since not all videos have an accurate description, Google is currently developing a game in which users are requested to describe the video shots. Google will then take an intersection of these descriptions, as there are bound to be similarities. If hundreds of thousands of people participate, it will be possible to obtain one single description for the video in which Google can look for relevant search commands.  Google’s method is entirely dependent on the descriptions made by the users themselves.’

Windmills

Gevers' method doesn’t depend on the verbal descriptive talents of users, but it does entail a vast amount of work. However, he doesn’t have to start from scratch. ‘We already have plenty of experience in object recognition; my group is very experienced at that. I was one of the first to develop a search engine for images (see link below), whereby I felt that colour could be very important.’ Previously, search engines made almost no use of colour, since colour is strongly dependent on the light source. An object seen in full sunlight shows a totally different colour than when seen in either fluorescent light or under a regular light bulb. Gevers has developed a programme that corrects for this difference. He has also tackled the problem of shape-invariance: the shape of a windmill varies greatly depending on the angle it is viewed from (see illustration below). Yet the computer must be able to identify a windmill in all these different shapes.

First of all, adjustments must be made for colour, and then the programme can look for important characteristics. ‘These consist of points where something interesting happens, e.g. a colour transition or a corner. It is here that information can be obtained, and so we place an ellipse around it (see illustration below), which also indicates the scale and orientation of the important information. Scale-independent characteristics are obtained by transforming the ellipse to a circle. We now have identically sized circles containing information. For example, one circle gives us the information that it contains some red and blue. We then count the incidence of circles with some red and blue. If we show the programme lots of images of a certain subject, it will learn which circles belong to that subject, and so it knows which points are important for that subject and can search for the correct points in all kinds of images.’

Showing the retrieved videos is also problematic: how can the user find the particular video shots he requires amongst a selection of retrieved videos shots? Gevers is also working on this problem. The set of frames used for the search also makes a big difference: it is, for example, easier to search in a set of news videos than in a set of home videos. ‘Compared to home videos, the news is very structured. It’s relatively easy to find a football match in the news: the green grass and the football are easy to identify. 91% of the images retrieved are indeed of football matches. The result is less successful with e.g. world leaders.’ There are, indeed, world leaders among the images Gevers shows us: more than half the images is correct. The other images also show many men in suits who are, however, not world leaders. The software has apparently used the characteristic ‘tie’ as a search criterion, although this is not only specific to world leaders, see figure. Gevers, apologetically: ‘But in this case only visual characteristics were taken into account.’