Iconclass and Sensory History

image

The visual sources of Sensory History

Seeing, hearing, smelling, tasting and feeling often have a visual component. Therefore the iconography of representations where the five senses play a part, however implicitly, also often has a sensory component. Participating in a two-day workshop about the olfactory aspect of historical sources confirmed the idea that our visual heritage is extremely rich in sensory information and connotations. That information can be made quite explicit in pictures, but more often it is implied and can only be exposed through the interpretation of complex visual data. Making that information available for historical research is therefore not an easy task.
Iconclass, the classification that is most widely used for cultural content, does not seem to do justice to the richess of sensory connotations. Olfactory or auditory aspects of biblical, mythological and historical stories go unmentioned or remain more or less implicit. In many cases they are not easily retrievable. It is easy, for instance, to tag representations of paradise with Iconclass concepts, but in its present state the Iconclass system does not provide olfactory information about its “lost smell”. Neither does it refer to the sense of touch that plays a part in the depiction of the institution of marriage, another pre-Expulsion theme. This variant of the iconography, with Adam and Eve holding - and touching - each other’s hand, has an undeniable tactile aspect.

image

In a similar way, Iconclass is often silent when sound is implied by a picture. Here is a random example, showing soldiers shooting at a pack of dogs attacking them. A picture of soldiers using firearms will not be retrieved by the implied auditory aspect of its iconography. Nor will it be retrieved by its olfactory aspect, for example by a reference to the smell of gunpowder, a result of the firing of rifles.

image

Thinking about ways to improve the performance of Iconclass when it comes to the sensory aspect of visual information, my first instinct was to focus on the tools the Iconclass system itself has on board. Those can be broadly subdivided into the words of the concept definitions, the keywords of the entry vocabulary, the cross references between concepts, and the references between related keywords.

image

In its present state, for example, the smell of the venison prepared for Isaac, the feeling of the goatskin Jacob has put on his arms, nor the blindness of Isaac, are made explicit in the Iconclass concept definition nor in the keywords with which you can find the concept.

If we want to alert researchers of sensory history to those aspects, we must also tag it with concepts located in a different branch of the Iconclass tree. The senses involved here – sight, smell and feeling – are found in the Iconclass category of the “five senses” 31A3 the (five) senses and they can be combined with the concept actually identifying the story of Jacob’s deceit
In this specific case it will be easy to change the definition to incorporate the sensory information, or add keywords like “smell” or “touch” to retrieve the concept.

However, surveying the content of the Iconclass schedules from this perspective, it dawned on me that there is a real risk that we shall overshoot our mark when we try to solve the problem with the internal instruments of Iconclass.
I had been looking for ways to enrich the Iconclass toolkit to improve its performance for olfactory information with the help of the information gathered by the Odeuropa team in an Excel sheet. As the list of concepts with potential sensory information grew and the web of cross references increased in density I realized that this approach could create so many references to the sensory aspect of our visual heritage, that to many users it would lose its meaning.
Iconclass contains some 40,000 concepts, but the combinatory explosion unleashed when we include the keys (secondary hierarchies) in the equation, will boost the tally up to some 1,5 million concepts.

So, even if only a small percentage of Iconclass concepts has a sensory connotation – and I have a feeling that the percentage will actually be quite significant – the number of hits that would be produced for words like smell, sound, touch, taste or sight would be massive. Massive, that is, if by adapting the Iconclass system itself searching with those words would always produce hits.

To grasp the consequences it might help to have a look at how Iconclass deals with the iconography of saints. Here is an example:


You do not have to be a specialist in Christian iconography to realize that not every depiction of saint Augustine will show the attributes mentioned in the concept definition. Still, if a user searches for “flaming heart” - included in the definition and in the keywords (in italics) - the Iconclass system will always respond with a reference to saint Augustine. To alert the user to the fact that this reference can be – and often will be –a false positive, we decided to add the word “possible” to the concept definition. The original printed format of Iconclass, of which this is a snippet, used italics for attributes. and relied on the assumption that a reader would know that not all representations of saint Augustine would not always show all attributes. In a computerized environment that is simply too implicit.

An Iconclass-wide adaptation of definitions and keywords with potential sensory content along these lines will therefore in all likelihood indeed overshoot its target.

In addition, the small army of cataloguers that have built the corpus of Iconclass-tagged images will not have taken into account that the images they were cataloguing may have sensory connotations. Those connotations will in most cases not have been perceived as relevant. So, to enrich the concepts they have used for the iconography of their collections with explicit references to sensory content would have an enormous impact on the information they thought they were providing when they selected the Iconclass concepts. We would be changing the rules midway through the game…

The adaptation of the vocabulary, in other words, must take into account that the vocabulary has been used for over fifty years and millions of images have been tagged with it.

So, our task is twofold:

  • To make it possible to alert researchers of Sensory History to aspects of iconography relevant to their research, taking into account that this may cover an extremely wide range of historical sources
  • To avoid creating so many references to sensory experiences that the information will be perceived as self-evident and trivial; from “smell” to “smoking tobacco” … duh…

The solution I am inviting you now to consider is a simple one, seen from the perspective of the basic Iconclass toolset. However, it is also dependent on the way Iconclass information is offered in local implementations of the system. This is a much more general issue: while all institutions using Iconclass information for their iconographic information by definition use a common vocabulary, the experience they offer to their end users can be wildly divergent, up to the point that it obscures the fact that they are all using the same vocabulary …

Keep that in mind when evaluating the following proposal.

Combining concepts

Even though in very specific cases, like the blessing of Jacob, it will be useful and easy to enrich the specific definition, the main focus of the adaptation of the schedules will be on category 31A3 the (five) senses.

In the appendix you will find the concepts now included in this category. It is easy to see that the category has not expanded to great detail. As a result there is ample room for new concepts. I have now added a handful of concepts – the ones in red – to give you an idea of where we could go with this. You will recognize the main menu of the Odeuropa tools site in the proposed new section for 31A333 olfactory concepts. I’ve provisionally copied that subdivision for sounds as well, to illustrate the principle for other senses as well.

Now, used in isolation these concepts do not contribute much. However, if they are combined in the tagging of a source with concepts from other parts of the schedules, they will have an effect.
image

An obvious example is provided by this Aesop illustration. In addition to the concepts identifying the scene as an Aesopian story, the picture could be tagged with the following concepts:
31A3332 smell sources
31A623 faeces, excrement
If so, a word like “smell” or “stench”, but also a phrase like “source of a smell” can be combined – at retrieval time – with “faeces”.
image

In a similar setup a picture such as this one of the Fall of Jericho could be tagged with
31A3332 sound sources
48C7352 horn, trumpet, cornet, trombone, tuba
Which would then lead – potentially - to searches like “sound” or “source of a sound” combined with “horn”.

Do note that because of the hierarchical construction of Iconclass, this would also mean that “sound” and “music” would then retrieve a picture of Jericho’s destruction.

I have deliberately used the word potentially because for this strategy to really benefit the end user of the information, two conditions must be met.

  • The cataloguer/researcher who decides that the sensory aspect of the visual sources deserves to be documented, must tag the picture with a combination of concepts. So, yes, it does require an effort.
  • The way searching with Iconclass is implemented must make it possible to retrieve records from a database by simultaneously searching for a combination of notations, or of words that belong to different definitions. (i.e. the Arkyves model…)

I have limited myself for now to the expansion of only two of the candidate concepts, i.e. 31A32 and 31A33. The actual expansion and phrasing (and translation!) would require input from specialists of Sensory History.

The proposed expansion is printed in red. The wording is copied from the online Odeuropa tools and adapted for sounds.

In blue and preceded by an arrow are suggested, hand-picked cross references. Again, just to give you an idea of what would be possible. The cross references at 31A31 sight, however, are already part of the online Iconclass system

Appendix



image

Mooie post.

Ik zou nog ergens expliciteren dat in sommige iconografieën en plaatsen geur, tast en geluid niet alleen bijkomstige elementen zijn, maar het verhaal juist betekenist geven/ centraal staan.

Het voorbeeld van Isaac die Jacob zegent: in het bijbelberhaal speelt tast een heel belangrijke rol omdat Isaac blind is. maar de reuk geeft in het verhaal de doorslag en die heeft nu juist weinig met het braadstuk te maken. het is het omgeknoopte jacchtvest van Esau waardoor Isaac aan de hand van geur denkt dat hij echt met Esau heeft te maken.

Verder heeft de geur soms juist bijna geen zichtbare component maar weet je dat het er is door verder onderzoek of kennis. Zoals Napoleon die standaard 4711 droeg bijvoorbeeld. Of de geur die opsteeg uit molens (bij ‘de kleine stinkmolen’ van Rembrandt verraadt de titel het tenminste nog).

In weer andere gevallen, zoals bij kasjmier sjaals, is soms niet bekend dat het een heel sterke geur had (namelijk patchouli).

Thanks Caro, useful comments.
I just saw your reference to the Nose first art historical scent wheel and could immediately identify concepts that should be added to Iconclass.
A first example would be the armpit which we could include as an extension of 31A2241 upper arm. I would propose to add it as a more specific term with the notation 31A22411.

Thank you for this marvellous discussion, Hans. You make a salient point about the cataloguers who were working on the images being unaware of the sensory connotations; Iconclass has existed substantially longer than sensory studies have.

And Caro makes a good point: certain sensory elements may have more significance than others. To add to this, when they are depicted in works of art, their significance may vary too. I would defer to Hans’ discussion of Augustine’s attributes: it can suggest possible sensory attributes, but cannot go as far as being prescriptive.

I am thrilled to read this proposal of cross-referencing, and that it may be applicable for all of the senses. Now a question – which may be relevant for all of the senses – I do imagine that there are certain scents that do not yet have Iconclass concepts, such as linen cabinet/ closet, civet, lily of the valley, jasmine, yet they are depicted in works of art. Would you still consider introducing concepts to Iconclass, which could then be cross-referenced? Would ‘armpit’ be the start of this?

Some examples

The Nose first art historical scent wheel quoted earlier, still has some open spots, where apparently no Iconclass equivalent could be found. I have made some provisional suggestions in the list below.
Of course this should be finalized in dialogue with experts in the field, but it should help to illustrate the point I wanted to make earlier. The concepts I propose as an extension of 31A33 smell, smelling (one of the five senses) e.g. 31A3332 smell sources and 31A3333 odour carriers should in actual cataloguing practice always be used in combination with other concepts.
Neither term is as yet an Iconclass concept, and before we add them we could use this forum to collect ideas about the best way to handle this.
Concepts like these will be enriched with keywords like “smell”, “source”, “odour”, “scent” to retrieve them from the system. When used in combination with one or two other concepts, like the ones in bold below - for shop and apothecary - that would create the option to do a search with these three words:
scent + apothecary + shop

If you want to get an idea how this would work in a pictorial information system where objects have been tagged with multiple Iconclass concepts, each with their own keywords, have a look at the results for a search for smell and flower at the Image browser at the Herzog August Library.
That functionality, however, is dependent on the way Iconclass is made to function as a retrieval tool instead of as a production tool.

But that does not mean that we cannot (should not) also enrich the production tool itself. For example:
a concept like armpit can be easily added. However, to identify it as a source of smell it would still be better to combine it with 31A3332 smell sources. That is a more flexible solution than adding “smell” as a default keyword reference to “armpit”.
Where Iconclass uses words that are a little outdated or unexpected - like carrion instead of cadaver - we can easily add the keyword, and thus increase the retrieval chances. We don’t have to create a new concept.

Below are some examples of terms that appeared problematic when the scent wheel was made.

Olfactory term Suggested Iconclass equivalent
armpit 31A22411 armpit (“child” of: 31A2241 upper arm)
tannery 47B251(+51) leather industry (+ preparing, cleaning, sorting, selecting ~ industrial processes)
cadaver 25F(+66) animals (+ carrion) add keyword “cadaver”
old vegetables 41C65 vegetables and fruit; still life of vegetables and fruit
We could add a key (+8) for “rotting” but also extend: 25G(+37) plants; vegetation (+ dying plant) as all vegetables are also plants and rotting is a process affecting also plants that are not vegetables
indole If the only way this is visualized is by white flowers, these could be used (in combination with one of the new extensions of 31A33):
25G41 flowers
22C4(WHITE) colours, pigments, and paints: white
bleaching field 41D422 bleaching (laundering)
25H17 meadow, pasture
47C131(+51) cleaning of material by bleaching (+ preparing, cleaning, sorting, selecting ~ industrial processes)
plague mask 31A4621 plague - could be extended with a new concept 31A46211 plague mask, or
41D283 mask
49G(+4) medicine, medical science (+ professional clothes of scholar, scientist) - this key could be extended to e.g. 49G(+41) medicine, medical science (+ clothes to protect against infections)
N.B. Visual Resources just published a special Covid issue about The Visual Cultures of the Virus
incense bowl 41C762 incense-burner ~ scents, perfumes
apothecary cabinet 46B13 shop, store
49G82 pharmacist, druggist, apothecary; pharmacy, drugstore, dispensing chemist
Aya Sophia 12I61 temple, shrine ~ Islam, Mohammedanism
11Q71 the place of worship and its equipment ~ Christian churches
61F(AYA SOPHIA) names of historical buildings, sites, streets, etc. (with NAME)
musk deer 25F24(MUSK-DEER) hoofed animals: musk-deer
civet cat 25F23(CIVET CAT) beasts of prey, predatory animals (with NAME)
street scene 25I141 street
guirlande 25G4111 garland, wreath
paper mill 47D31 windmill
47D41 watermill
49L641 paper (writing material)

Excellent. I’m passing this over to the Odeuropa team for comment. Thank you again!

Thanks Lizzie. It would be interesting if they could also develop ideas about how the suggested expansion:

  • 31A333 olfactory concepts
    • 31A3331 smells
    • 31A3332 smell sources
    • 31A3333 odour carriers
    • 31A3334 fragrant spaces

would relate to the “Principal Odor Map” that was introduced in the Google AI Blog last Tuesday.
While the bio-medical domain is the obvious first target for this type of research, its focus on smell sources and odour carriers seems to confirm that adding these concepts to a classification for cultural content is a good idea too…

In addition we should probably also add a concept to tag the “consumer” of a smell (for lack of a better word). Adding it to the list at 31A33 would allow it to be combined with all possible actors in the Iconclass system, human, animal (as illustrated below…) and even mechanical…

image image