Commentary on Todorovic's: Lightness and Illumination: Comments on Kingdom and Blakeslee and McCourt

Our chapter, entitled A Multiscale Spatial Filtering Account of Brightness Phenomena, initially considered the inverse problem in brightness perception, that is, how and when is the visual system able to recover surface reflectance and illuminance from the proximal luminance signal present at the photoreceptors. The luminance distribution falling on the photoreceptor array, and the neural activity this engenders, is the product of the reflectance of objects in the environment and their illuminance. The recovery of surface reflectance and illuminance information from the raw luminance signal is thus an ill-posed problem, in that there are myriad combinations of illumination and reflectance that can give rise to any particular intensity distribution, and in the absence of additional information there is no algorithmic solution. We know from our everyday experience, however, that the visual system does manage to solve the inverse problem in a satisfactory manner much of the time. The heuristic used by the visual system must, therefore, involve the use of additional sources of information and/or assumptions. Indeed, much of the debate in brightness perception, including the papers presented in this volume, centers on discovering the mechanisms the visual system uses to solve this problem.

The definitional issues regarding the terms lightness and brightness, also addressed in our chapter, are a logical outgrowth of the inverse problem. Todorovic, in his comments on our chapter (Lightness and Illumination: Comments on Kingdom and Blakeslee & McCourt), defended the use of the Trieste group definitions of lightness and brightness for their non-theoretical descriptive simplicity and for their logical agreement with the tasks given to subjects in some experiments. We agree that the description of the psychophysical variables of luminance, luminous reflectance and illuminance, and their perceptual correlation with brightness (perceived luminance), lightness (perceived reflectance) and perceived illuminance, is useful and conceptually valid. Our concern, however, is that the term lightness (defined by the Trieste group simply as perceived reflectance) encompasses too many different types of achromatic judgments. This is the direct result of the fact that, in its heuristic solution to the inverse problem, the visual system uses different strategies to estimate surface reflectance in different situations. For example, depending on the stimulus conditions (and subject instructions), lightness may refer to judgments that are identical to brightness judgments, that are identical to brightness-contrast judgments, or that represent an independent third dimension of achromatic experience (Arend and Spehar, 1993 a, b). Critically, this third dimension exists only when the stimulus contains a visible illumination component, i.e., only when the illumination across regions of the display is visibly non-uniform. This is the type of lightness judgment that we call inferred-lightness, because it requires that the subject deliberately take account of the illuminant to judge perceived reflectance. In other words, unlike the other two types of lightness judgments, inferred-lightness is not a direct judgment of surface color.

Complicating matters further, the Trieste definition of lightness is not the only definition of lightness currently in use. Sewall and Wooten (1991) and Dalby, Saillant and Wooten (1995), for example, define lightness specifically in terms of surface color (ranging from black to white). In their experiments subjects judged lightness using an absolute-judgment paradigm in which they reported the percentage of blackness perceived. This use of the term lightness is consistent with the definitions of both the Committee on Colorimetry of the Optical Society of America (1963) and the CIE (1970). Furthermore, contrary to Todorovic's assertions, this definition has been used to formulate subject instructions. The task the subjects were instructed to perform in order to obtain lightness judgments (i.e., rating the percent blackness of a stimulus) accords perfectly with this definition of lightness, and requires a subject to make an absolute judgment relative to white [footnote1]. Importantly, however, these instructions were deliberately construed to emphasize the achromatic color (black to white) of a stimulus, and to ensure that lightness judgments were not based on inference. Indeed, Sewall and Wooten (1991) and Dalby Saillant and Wooten (1995) argue that lightness instructions such as those used by Arend and Goldstein (1987), Schirillo, Reeves and Arend (1990) and Arend and Spehar (1993 a, b), in which subjects are asked to match the test patch to the standard "as if it were cut from the same piece of paper", encourage responses based on inference rather than perceived surface color.

The fact that lightness judgments are based on different information under different conditions, and that the various definitions of the term lightness refer to all or only a subset of these different types of lightness judgments, has created confusion in the literature. Thus, "lightness" data generated in one study may or may not be comparable or relevant to what other investigators study using the same name. This is demonstrated most clearly in the studies of Arend and Spehar (1993 a, b). They demonstrated three qualitatively and quantitatively distinct dimensions of achromatic experience: brightness (perceived luminance), local brightness contrast (differences in brightness), and lightness (perceived reflectance). Again, it is important to understand that Arend and Spehar's data indicate that all three dimensions existed only for sufficiently complex stimuli that included a visible illumination component. Under other, more impoverished, stimulus conditions only two dimensions of achromatic experience, brightness and brightness contrast, were available for matching, and these authors found that lightness matches collapsed onto one or the other of these two judgments depending on the stimulus conditions. For example, since under homogeneous illumination (even for complex stimuli) equal reflectances have equal luminances, it is not surprising that lightness matches collapse onto brightness matches. Arend and Spehar's (1993 a, b) data make it clear that it is irrelevant in this instance whether the investigator instructs the subject to match brightness (perceived intensity or luminance) or lightness (perceived reflectance) - the data generated will be identical.

Todorovic argues that the distinction between lightness and brightness is still conceptually valid in this situation, although he does admit that it "may not make much of a difference" and that "a general, undifferentiated notion of perceived achromatic color may suffice." We suggest that by recognizing the perfect correlation between lightness and brightness under conditions of homogeneous illumination (real or simulated), and by using another term (or a qualifier of the term lightness), we may alleviate the confusion stemming from the fact that studies of "lightness" under these conditions are comparable to other studies of brightness but may not, confusingly, be comparable to other lightness studies. For instance, Arend and Spehar's (1993 a, b) data inform us that these lightness data might be comparable to lightness data obtained in ambiguous center-surround displays, but only if subjects interpret the intensity difference between the surrounds of the test and standard patches to have resulted from differential reflectance. If, however, subjects interpret the intensity difference to have resulted from differential illumination, then lightness judgments are identical to brightness-contrast judgments and the data may not be comparable. Likewise, lightness judgments that refer to the third dimension of achromatic experience, distinguishable from both brightness and brightness contrast, are not comparable with either of the other types of lightness judgment.

In summary, although lightness defined as perceived reflectance is conceptually simple and useful, it has been demonstrated experimentally that one term, lightness, can refer to three very different types of judgments that are not comparable. At the very least we should be mindful of this fact and insist that sufficient experimental detail is always available for readers to make the necessary distinctions. Modifications of the term lightness might be useful in this regard, and need not change the general conceptual definition of lightness as perceived reflectance. Modifiers would simply clarify that the term lightness encompasses three distinct and nonequivalent types of judgments that occur under different stimulus conditions. We introduced the term inferred-lightness in our Chapter to specify the third dimension of achromatic experience. We also suggested that brightness-lightness (or b-lightness) and brightness-contrast lightness (or bc-lightness) might be useful modifiers.

Todorovic commented that basic concepts are preferably defined in a non-theoretical manner. We used the term inferred-lightness to describe lightness judgments representing the third dimension of achromatic experience because this term communicates the essential concept of a cognitive interpretation or appraisal of a stimulus property rather than one based on a primary, readily accessible, sensory quality (such as brightness or brightness-contrast). We find this labeling useful, but recognize that others might prefer a different nomenclature.

Our use of the term inferred-lightness may also be somewhat confusing since, in another sense, all three types of lightness judgments are the outcome of perceptual inference. Considering the fact that reflectance is always confounded with illuminance, in that they jointly produce the luminance signal on the retina, even lightness judgments based on brightness or brightness contrast involve a strategy or assumption on the part of the observer (conscious or unconscious) for estimating illuminance. Under conditions in which the term inferred-lightness is appropriate, however, we argue that the "visibility" of the illumination component demands that the observer consciously take note of and discount the illumination in order to render a perceived reflectance judgment. As we discuss below this may, in some instances, be a fairly simple task or may, in other circumstances, require a more deliberate and effortful type of mental calculation.

Todorovic argues that it is not easy to substantiate the immediacy of brightness judgments (viz., that they are primary sensations) or the presence of cognitive interpretations in lightness judgments (viz., that they are the outcome of a perceptual inference). We feel that the commonly used distinction between sensation and perception is useful and need not imply a rigid division. Sensations commonly refer to immediate, fundamental, and direct experiences of the physical environment, usually produced by simple, isolated physical stimuli. Perceptions, on the other hand, typically involve cognitive processes in which meaning, relationships, context, judgment, past experience and memory play a role. Although it is true that no clear separation between sensation and perception may be apparent in natural viewing, we would argue that carefully controlled laboratory experiments have separated what we mean by sensation and perception, at least at the extremes of the continuum. For example, brightness (perceived intensity) is commonly cited as an exemplar of a sensation because it corresponds to the conscious awareness of one of the simplest attributes of a visual stimulus, namely, its intensity. It is in this sense that we argue that brightness (perceived luminance) is directly given, since luminance is the visual stimulus. Indeed, the simplest visual stimuli, unrelated colors, are described completely by their hue, saturation and brightness. When they are achromatic, these stimuli vary only in brightness; they may appear white but have neither a gray nor a black component (Pokorny, Shevell and Smith, 1991). Since adaptation and lateral inhibition originate at the earliest levels of processing in the retina, it is true that brightness is a fundamentally relative measure. Nevertheless, it is this basic relative brightness information, or map, that we argue is used to make both brightness and brightness contrast judgments (and therefore b-lightness and bc-lightness judgments), as well as inferred-lightness judgments.

That brightness is a primary sensory quality, and that we have direct access to this brightness map, we feel is also the most parsimonious assumption. Since reflectance and illuminance are confounded in the proximal luminance stimulus, it seems reasonable to assume that their values must be assigned based on strategies and/or assumptions involving information derived primarily from the brightness map itself. Although possible, it seems counterintuitive to accept, in the absence of direct supporting evidence, the situation proposed by Todorovic, viz., that "lightness judgments may, at least in principle, be based on a 'direct' readout of a Gibsonian higher-order variable (the luminance ratio is a simple example), whereas conscious brightness judgment, being an unusual and novel task in everyday conditions, may involve indirect inferences based on perceived reflectance and illumination." Note that we really only disagree in a definitional sense with the first statement that "lightness judgments may, at least in principle, be based on a 'direct' readout of a Gibsonian higher-order variable (the luminance ratio is a simple example)". Indeed, the work of Arend and Spehar (1993 a, b) has led us to argue that lightness is, in some instances, in fact based on brightness-contrast. We don't, however, see any explanatory utility in calling this a "Gibsonian higher-order variable". We are, however, in fundamental disagreement with the second statement that "conscious brightness judgment, being an unusual and novel task in everyday conditions, may involve indirect inferences based on perceived reflectance and illumination". We submit that conscious brightness judgments are neither unusual nor novel tasks in everyday conditions. Indeed, certain types of "signature" brightness variations such as shadow, highlight, shading and specularity, are precisely what allow us to detect and represent illumination itself. In other words, these differences in brightness across space actually constitute the "visible illumination component" that is a crucial determinant in creating the independent dimension of inferred-lightness, and in deriving object shape from shading.

Another argument favoring the interpretation that brightness judgments are based on primary sensations can be made on the basis that they are, relative to inferred- lightness judgments, extremely easy to make. The observation that lightness judgments may also sometimes appear effortless may reflect, again, the fact that lightness judgments are often based on, or are equivalent to, judgments of brightness or brightness contrast. Even under naturalistic conditions where a clear illumination component is visible (and where inferred-lightness is therefore truly separable from brightness and brightness contrast) there are often abundant cues available that easily disclose the lightness values of surfaces without requiring an observer to calculate them (calculation being a much more difficult cognitive task). Consider the case of a cast shadow (see Kingdom's chapter this volume, Fig. 11) falling across a surface, where the brightness of the shadowed surface is clearly lower than that of the more highly illuminated region. By knowing or assuming that the surface is homogeneous an observer can quickly and effortlessly judge (or match) the lightness of the shadowed region by simply identifying it with the contiguous non-shadowed region. However, when a visible illumination boundary does not cross the surface whose lightness is to be judged, for example, when that surface lies completely beneath and within the shadowed region, the judgment of inferred lightness becomes more effortful and requires a deliberate attempt to indirectly discount the illuminant by determining its effect on other remote surfaces in the scene. Inferred-lightness determined under these conditions is not an easy task, and it becomes clear that inferred-lightness does not fall under the usual definition of a sensory experience. Dalby, Saillant and Wooten (1995) argued that the results of Gilchrist (1977; 1980), as well as their replication by Schirillo, Reeves and Arend (1990), showing an effect of perceived depth on lightness judgments, were probably referring to the outcome of such an explicitly cognitive, and not sensory, task. For example, Dalby, et al., (1995) argue that: "Their 'lightness match' is apparently an explicit cognitive task that amounts to inferences about actual physical reflectance (i.e., 'cut from the same piece of paper'). Such a task would, of course, involve some estimation about the illumination, which is where examining the entire display may enter the picture". It is interesting that Schirillo, et al., (1990) also noted that "All subjects reported brightness matches as being easier than lightness matches, and felt more confident about their final settings in the case of brightness. This was reflected in slightly smaller standard errors for brightness matches than for lightness matches. In all three experiments, brightness matches also took far less time to make than did lightness matches" [footnote 2].

Blakeslee and McCourt's commentary on Kingdom's: Comments on Alan Gilchrist's talk at the York 2001 Conference

Interestingly, the same definitional issues discussed above regarding brightness and lightness arise in connection with a demonstration presented at the 2001 York conference by Alan Gilchrist, and described in Kingdom's comments on the talk (see Fred Kingdom's comment on Alan Gilchrist's talk at the York 2001 conference, Fig. 4a). Here a SBC-like stimulus, a card consisting of two test patches on two backgrounds, was presented in which the left half of the card was brightly illuminated by a theatre lamp from below. It was clear that the left half of the card was brightly lit, since the lamp itself and a penumbra between the two halves of the card were clearly visible. The backdrop against which the entire card was seen was illuminated only by room light. Under these conditions the two test patches were described to be about equal in lightness (what we will argue is actually better described by the term brightness). When the lamp was switched off, it was clear that the two test patches were actually on the same black background, and that the left patch now appeared much darker than the right patch. Quoting Kingdom "According to Alan, this showed that our visual system had failed to take into account the lamp's illumination when estimating the lightness of the left patch, even though it was obvious that it was more brightly illuminated than the patch on the right. If the visual system were able to segment the scene into its intrinsic images, i.e., its illumination and reflective components, then we would have discerned the true lightness relationship between the patches." Kingdom argues that this demonstration is not sufficient to reject the idea that intrinsic-image processing forms an important component of lightness perception because "Alan's demonstration simply does not isolate the mechanisms involved." He further argues that "Our visual system may have partially discounted the lamp's illumination when ascribing the lightness to the patch, but the effect of any discounting was almost certainly overwhelmed by the effect of the luminance contrast between the left side of the card and its background." Our own interpretation of this demonstration is somewhat different from both Gilchrist's and Kingdom's. In Gilchrist's first display, in which the theatre lamp is on, an illumination component is clearly visible. Thus, according to the work of Arend and Spehar (1993a,b), this stimulus is one in which it should be possible to infer the lightness (perceived reflectance) of the test patches, and this lightness judgment should be different from the brightness judgments. We argue that one can indeed estimate the lightness of the test patches in this display but that, because the illumination boundary does not cross the test patch, this judgment is of the more effortful cognitive variety discussed earlier, and is not sensory nor in any sense automatic. In other words, what appears similar in this phase of the demonstration is actually the brightness (perceived intensity) of the test patches, and this information is immediately available. But, if instructed to do so, and given enough time, observers could certainly make a mental calculation of the illumination from information provided by other parts of the scene, and discount it from the test patches or the background cards to achieve an estimate of their lightness (perceived reflectance). Although observers might not be particularly accurate in their estimates, observers with reasonable powers of deduction would definitely conclude that the patch on the left must be darker (less reflective) than the patch on the right. We further argue that asserting that the two patches looked more-or-less equal in "lightness" when the illumination was present represents a truly erroneous use of this term according to the perceived reflectance definition. Given the obvious inhomogeneity of illumination it is a logical impossibility for these two surfaces to share a common reflectance. They can, however, share a common intensity, and that is why they appear equally bright.

Commentary on Gilchrist's: Comments on Fred Kingdom's Comments on Gilchrist's chapter

We would like to dispute a point concerning Gilchrist's explanation of grating induction according to his anchoring model. In Fred Kingdom's comments on Alan Gilchrist's talk at the York 2001 conference, he notes that the perceptual grouping within the GI stimulus necessary to predict the effect in terms of anchoring appears arbitrary. We agree. Gilchrist, as part of his defense of the anchoring explanation (see Comments on Fred Kingdom's comments on my talk), notes that, according to anchoring theory, the homogeneity of the GI test field should make the test field appear homogeneous in lightness (i.e., brightness). Gilchrist argues that anchoring predicts weaker induction in the classic GI stimulus, where the test field is continuous, compared to a display in which the test field is broken and asserts that this is what Blakeslee and McCourt show in their Fig. 2.1 (this volume). This assertion is, in fact, incorrect. Blakeslee and McCourt (1997) used a point-by-point brightness matching task to assess differences in the structure and/or magnitude of GI as a function of changing test field height and width. They found no discontinuities or changes in either the structure or magnitude of induction as the test field was transformed from the standard GI configuration (32^o continuous test field) to the elongated but separate test field conditions (14^o and 12^o). Although it is true that as test field width decreased below 12^o the magnitude of induction began to increase, this effect is entirely explained by the recruitment of smaller scale filters in a multiscale spatial filtering model. According to Gilchrist's account there should be an immediate increase in the magnitude of induction as soon as the test field is broken (similar to the effect of placing a thread across the Benussi ring stimulus). Since this is not the case one must therefore conclude that these data argue against (not for) an anchoring explanation of GI.

Commentary on Gilchrist and Economou: Dualistic Versus Monistic Accounts of Lightness Perception

We view explanations of brightness perception in terms of multiscale spatial filtering, and the ODOG model in particular, as modern versions of lateral-inhibition models. The ODOG model differs significantly from traditional (textbook) explanations of lateral inhibition, however, in that such treatments rarely describe multiscale arrays, but instead refer to single spatial filters tuned to relatively high spatial frequencies. The responses of small filters are, of course, restricted to spatial regions of stimuli that contain high spatial frequencies, such as edges, and this is the origin of what we regard as the erroneous notion that lateral inhibition is merely a mechanism for encoding or enhancing edges. If the term "lateral-inhibition" is impoverished in this manner to refer to a single-scale high-frequency or edge encoding mechanism, then we agree with Gilchrist and Economou that this form of lateral-inhibition is inadequate to explain brightness since "by this account, the first level produces edge signals, not lightness levels ... and one needs a whole theory of lightness to get from edge signals to lightness levels." Interestingly, however, Gilchrist and Economou note "one can say that lateral-inhibition produces lightness levels if one accepts a point-wise conception of lateral-inhibition, rather than an edge-encoding conception. Such a point-wise conception is implicit in the familiar scalloped profiles of the simultaneous contrast display after it passes through a lateral inhibitory network (Cornsweet, 1970). But the absence of scallops predicted to occur in homogeneous surfaces has led most students of lightness to abandon the point-wise conception in favor of the edge-encoding conception." Our version of a multiscale filtering approach embodies just such a point-wise conception of lateral-inhibition, and the output of the ODOG model produces a brightness map that accounts with remarkable accuracy for a wide variety of diverse brightness effects. It is important to note that within the multiscale array it is the output of relatively large filters that principally determines the value(s) of the brightness map in the central region(s) of large uniform test fields. Furthermore, contrary to the assertion of Gilchrist & Economou, structured brightness profiles are a well-documented feature of numerous brightness displays. The familiar scalloping seen in the Chevreul (or staircase) illusion is a clear example. In addition, studies using point-by-point brightness matching have quantified clear structure in the brightness matching profiles of both SBC (Heinemann, 1972; Blakeslee and McCourt, 1997) and GI displays (Blakeslee and McCourt, 1997). Blakeslee and McCourt (1997) found that for GI stimuli (continuous 32^o wide test fields) in which the inducing field consisted of one cycle of a sinewave grating, the induced grating was well fit by a sinusoid. Sinusoidal variation also characterized induction in the split test field (SBC) conditions (two 14^o, 12^o and 8^o wide test fields) at three different test field heights (1^o, 3^o, and 6^o). Interestingly, for test fields narrower than 8^o in width the structure of induction in the test field flattened, and eventually showed the familiar concave cusping (scalloping) for test field widths of approximately 3^o. In preliminary modeling attempts Blakeslee and McCourt (1997) found that the changes in the pattern of brightness across the test field, as well as the stimuli in which these changes occurred, depended on the relative weighting of the filters across scale. Blakeslee and McCourt (1997) therefore used the quantitative brightness profile data to determine the optimal weighting function by which to combine the multiscale filters in the original DOG model. Combining filter outputs across spatial scale such that weight increased with center frequency as a power function with a slope of 0.1 was found to be optimal, and clearly predicted the brightness profiles. This weighting is consistent with the shallow low-frequency fall-off of the suprathreshold CSF. Blakeslee and McCourt (1999) also clearly demonstrated (and the ODOG model predicted) scalloping in the test patches of White's (1979) effect (see Fig 2.9 a-c). Indeed, we argue that only multiscale spatial filtering can easily account for these types of induced brightness gradients.