Artificial Visual System for Orientation Detection

Abstract

The human visual system is one of the most important components of the nervous system, responsible for visual perception. The research on orientation detection, in which neurons of the visual cortex respond only to a line stimulus in a particular orientation, is an important driving force of computer vision and biological vision. However, the principle underlying orientation detection remains a mystery. In order to solve this mystery, we first propose a completely new mechanism that explains planar orientation detection in a quantitative manner. First, we assume that there are planar orientation-detective neurons which respond only to a particular planar orientation locally and that these neurons detect local planar orientation information based on nonlinear interactions that take place on the dendrites. Then, we propose an implementation of these local planar orientation-detective neurons based on their dendritic computations, use them to extract the local planar orientation information, and infer the global planar orientation information from the local planar orientation information. Furthermore, based on this mechanism, we propose an artificial visual system (AVS) for planar orientation detection and other visual information processing. In order to prove the effectiveness of our mechanism and the AVS, we conducted a series of experiments on rectangular images which included rectangles of various sizes, shapes and positions. Computer simulations show that the mechanism can perfectly perform planar orientation detection regardless of their sizes, shapes and positions in all experiments. Furthermore, we compared the performance of both AVS and a traditional convolution neural network (CNN) on planar orientation detection and found that AVS completely outperformed CNN in planar orientation detection in terms of identification accuracy, noise resistance, computation and learning cost, hardware implementation and reasonability.

1. Introduction

In 1981, David Hubel and Torsten Wiesel won the Nobel Prize in Medicine because of their landmark discovery of orientation preference and related works [1,2]. Based on this remarkable discovery, Hubel and Wiesel found the orientation-selective cells in the primary visual cortex (V1) and proposed a simple yet powerful model of how such orientation selectivity could emerge from nonselective thalamocortical inputs [1]. The model has become a central frame of reference for understanding cortical computation and its underlying mechanisms [3,4,5]. Gaining more and more insight into the functional mechanisms of the visual cortex may bring the capabilities of artificial vision closer to those of biological systems and result in new developments of computer architectures. However, despite 60 years of intense research effort, three basic questions are still unanswered [6,7,8]: (1) how, (2) to what degree, and (3) by what mechanisms do the orientation-selective cells contribute to the detection of the global orientation of an object with different sizes or positions? In this paper, we first offer a novel quantitative mechanism to provide an explanation for how selectivity for planar orientation could be produced by a model with circuitry that is based on the anatomy of the V1 cortex and how the selectivity for planar orientation contributes to the detection of the global planar orientation of a rectangular object with different sizes, shapes or positions. We assume that planar orientation-selective cells—which we call local planar orientation-detective neurons—exist in in the retina of the visual nervous system. Each of these local planar orientation-detective neurons receives its own input through photoreceptors and ON-OFF response cells from the receptive field, picks up selectively an adjacent input, and computes a response only to the planar orientation from the selected adjacent input. We implement a model of the local planar orientation-detective neuron based on the dendritic neuron model that the authors proposed previously [9,10,11] and use it to realize several planar orientation-detective neuron models, each of which responds only to a particular planar orientation. Then, we propose four possible schemes to measure the activation of the local planar orientation-detective neurons: (1) scanning over two-dimensional inputs of an image, and for every input convolving their adjacent inputs with a local planar orientation-detective neuron; (2) scanning over two-dimensional inputs of an image with a group of local planar orientation-detective neurons, (3) sliding over two-dimensional inputs of an image with a small array of a grouped local planar orientation-detective neurons and (4) letting each input of a two-dimensional image follow its own independent local planar orientation-detective neurons. Since these neurons give local planar orientation responses that are localized in spacem and these neurons’ outputs can be taken as evidence about the global planar orientation, we can thus obtain the global planar orientation directly by measuring the outputs of these local planar orientation-detective neurons. Secondly, based on this mechanism, we propose an artificial visual system (AVS) for planar orientation detection and other visual information processing. To prove the effectiveness of our mechanism and the AVS based on this mechanism, we conducted a series of experiments using a dataset of a total of 20,000 images of rectangular objects with various sizes and positions at many different planar orientations. Computer simulations show that the mechanism and the mechanism-based AVS performed the detection of planar orientation very accurately in all experiments regardless of the sizes, shapes and positions of objects. Furthermore, we used a traditional convolution neural network (CNN), trained it to perform planar orientation detection and compared its results with those of the AVS. Based on the computer simulations and analysis, we conclude that the AVS outperforms the CNN in planar orientation detection in terms of identification accuracy, noise resistance, computation and learning cost and reasonability.

2. Methods

2.1. Dendritic Neuron Model

Artificial neural networks (ANNs) have been a research hotspot in the field of artificial intelligence since the 1980s [12,13]. An ANN is a mathematical model which mimics the information processing mechanism of the synaptic connection structure in the brain. To date, hundreds of models of artificial neural networks have been developed, and they have shown a very good performance in technical fields such as pattern recognition, medical diagnosis and time-series forecasting [13,14,15]. However, all these networks have used the traditional McCulloch & Pitts neuron model for their basic computation units [16]. This McCulloch & Pitts model does not take the nonlinear mechanisms of dendrites into account [17]. Meanwhile, recent studies have provided strong circumstantial support for the notion that dendrites play a key role in the overall computation performed by a neuron [18,19,20,21,22,23,24,25]. Koch, Poggio and Torre found that in the dendrites of a retinal ganglion cell, if an activated inhibitory synapse is closer than an excitatory synapse to the cell body, the excitatory synapse will be intercepted [26,27]. Thus, the interaction between the synapses on dendritic branches can be considered as a logical AND operation [28], and a dendritic branch point may sum currents from the dendritic branches, such that its output would be a logical OR on its inputs [29,30,31]. The signal is then conducted to the cell body (soma), and when it exceeds the threshold, the cell fires, sending a signal down the axon to other neurons. Figure 1a shows a model that implements an idealized $\delta$ cell. Here, if the inhibitory interaction is described as a NOT gate, the operation implemented in Figure 1a could be read as:

$$Output=X_1{X}_2+\overline{X_3}{X}_4+\overline{X_5}{X}_6{X}_2$$

(1)

where $X_1$, $X_2$, $X_4$ and $X_6$ denote excitatory inputs, and $X_3$ and $X_5$ represent inhibitory inputs. Each input is either a logical zero or one. Thus, the signal to the cell body (soma) becomes 1 when and only when $X_1=1$ and $X_2=1$, or $X_3=0$ and $X_4=1$ or $X_5=0$ and $X_6=1$ and $X_2=1$. Furthermore, the $\gamma$ cell receives excitatory and inhibitory synapses distributed from the tip to the soma, as shown in Figure 1b, thus reading,

$$Output=\overline{X_1}{X}_2{X}_3$$

(2)

Several experimental examples, such as direction selectivity in retinal ganglion cells [32] and coincidence detection in the auditory system [33], have provided strong circumstantial support to Koch’s model [27]. By taking the nonlinearity of synapses and nonlinear interaction among these synapses into consideration, researchers proposed a learnable dendritic neuron model (DNM) [9,10,11]. The DNM was successfully applied to many burning questions, such as liver disorders analysis, breast cancer classification, and financial time series prediction [34,35,36,37].

2.2. Local Planar Orientation-Detective Neuron

In this section, we describe the structure of DNM in detail for orientation detection. For simplicity, we only consider the composition of four neurons for orientation detection. Usually, the receptive field can be divided into two-dimensional $M\times N$ regions. Each region corresponds to a minimal visible region. For simplicity, we consider binary images. When light falls on a region, the electrical signal—for example, one—is transferred through its photoreceptor and ON-OFF response cells to ganglion cells and the ganglion cells perform various visual information processing steps [38]. Of course, by introducing horizontal cells, grayscale images and color images can also be treated easily. Here, we assume that there are simple neurons that can detect the specific orientation of a line. The input signal of region $(i,j)$ is expressed by $X_{i,j}$, where i and j correspond to position in the two-dimensional receptive field. Thus, for an input signal $X_{i,j}$, if we use the local orientation-detective neurons and consider eight regions adjacent to $X_{i,j}$ only, we can implement four local orientation-detective neurons by picking up selectively a particular region in that direction.

Figure 2 shows an idealized $\gamma$ cell for zero-degree planar orientation detection. What we need to consider is only the $X_{i,j}$, $X_{i-1,j}$ and $X_{i+1,j}$. If and only if $X_{i,j}$, $X_{i-1,j}$ and $X_{i+1,j}$ are all equal to 1, the $\gamma$ neuron is activated and the output of the soma is equal to 1. Similarly, the local planar orientation-detective neurons for other planar orientations can also be implemented.

Figure 3 shows the four structures of the local planar orientation-detective neurons. For example, the 45 degree detective neuron at region $(i,j)$ has different input signals from adjacent inputs $X_{i-1,j+1}$ and $X_{i+1,j-1}$ besides $X_{i,j}$, the inputs to the 90 degree detective (vertical detection) neuron at i, j come from $X_{i,j-1}$ and $X_{i,j+1}$ besides $X_{i,j}$, and the inputs to the 135 degree detective neuron at region $(i,j)$ are set to $X_{i-1,j-1}$, $X_{i,j}$ and $X_{i+1,j+1}$. Therefore, we can ensure that all planar orientation-detective neurons can be realized by $\gamma$-like cells. For simplicity, the size of the window (pixel matrix) is $3\times 3$, so we can only select these four planar orientations. If the size of the window increases, more planar orientations can be detected. For example, a $5\times 5$ window could detect 8 planar orientations, and so on.

2.3. Global Planar Orientation Detection

As mentioned above, the local planar orientation-detective neurons in the visual system respond by performing an interaction of the effect of light falling on their receptive field, for example, a local planar orientation-detective neuron extracts simple information on one planar orientation at one position in the receptive field by interacting with the input of the position from photoreceptors and ON-OFF response cells with its neighboring inputs. Here, we assume that the information of the local planar orientation can be used for judging global planar orientation. Thus, we can merely measure the strength of the activities of all local planar orientation-detective neurons over the receptive field (for example, the number of fired neurons) and derive a judgement as to planar orientation by summing the neurons’ outputs in different planar orientations respectively and taking the planar orientation with the maximal one as the global planar orientation. In order to measure the strength of the activities of the local planar orientation-detective neurons, with a total of $M\times N\times 4$ local planar orientation information, for a two-dimensional receptive field ($M\times N$), we have four possible schemes:

• One-neuron scheme: we assume that there is only one local planar orientation-detective neuron available and the local planar orientation-detective neuron is used to scan every region $(i,j)$ for $i=1,2,\dots ,M$ and $j=1,2,\dots ,N$ over the two-dimensional receptive field ($M\times N$), and at every position scans two adjacent positions at one direction, covering four directions in total, thus yielding $M\times N\times 4$ local planar orientation information;
• Multi-neuron scheme: we assume, for simplicity, that there are four local planar orientation-detective neurons, and that they are used to scan every region $(i,j)$ for $i=1,2,\dots ,M$ and $j=1,2,\dots ,N$ over the two-dimensional receptive field ($M\times N$), thus yielding $M\times N\times 4$ pieces of local planar orientation information;
• Neuron-array scheme: we assume, for simplicity, that there are four local planar orientation-detective neurons that are arrayed in $m\times n$ ($m<M$, and $n<N$), and that the arrayed neurons slide over the two-dimensional receptive field ($M\times N$) without overlapping, thus yielding $M\times N\times 4$ pieces of local planar orientation information;
• Full-neuron scheme: we assume that every input corresponding to the region ($i,j$) of a two-dimensional receptive field ($M\times N$) has its own local planar orientation-detective neuron. That is to say that there are $M\times N\times 4$ local planar orientation-detective neurons. Thus, within the local receptive field, the local planar orientation-detective neurons can extract elementary local planar orientation information. The local planar orientation information is then used to judge the global planar orientation. In order to help the understanding of the mechanism with which the system performs planar orientation detection, we used a simple two-dimensional ($5\times 5$) image of a bar in 45 degrees, as shown in Figure 4. Without the loss of generality, we use the four-neuron scheme in which the four local planar orientation-detective neurons scan every position from (1, 1) to (5, 5) over the two-dimensional receptive field ($5\times 5$), and yield the local planar orientation of the positions.

2.4. Artificial Visual System (AVS)

The visual system comprises the sensory organ (the eyes), the connecting pathways through to the visual cortex and other parts of central nervous system. As mentioned above, in the visual system, the local visual feature-detective neurons such as local planar orientation-detective neurons can extract elementary local visual features such as local planar orientation information. These features are then combined by the subsequent layers in order to detect higher-order features, for example, the global planar orientation of an object. Based on this mechanism, we have developed a generalized artificial visual system (AVS), as shown in Figure 5. Neurons in layer 1 (also called the local feature-detective neuron (LFDN) layer), corresponding to neurons in the V1 cortical area, such as the local planar orientation-detective neurons, extract elementary local visual features, for example, the local planar orientation information. These features are then sent to the subsequent layers, corresponding to the middle temporal (MT) area of the primate brain, (also called the global feature-detective neuron (GFDN) layers) in order to detect higher-order features, for example, the global planar orientation of an object. Neurons in layers can be expressed simply as the summation of the outputs of neurons from layer 1, for example, for planar orientation detection, motion direction detection, motion speed detection and the perception of binocular vision; or one layer; or two layers corresponding to V4 and V6; or three layers corresponding to V2, V3 and V5; or even a multi-layer network, for example, for pattern recognition. It is worth noting that the AVS is a feedforward neural network, and any feedforward neural network can be trained by means of the error backpropagation method. The difference between AVS and traditional multi-layer neural networks and convolutional neural networks is that the local feature detective neurons (LFDNs) in layer 1 of the AVS can be designed in advance according to our prior knowledge, for example, how many neurons and what kind of neurons are needed, and in most cases they do not need to undergo the learning process. Even if learning is needed, learning with the AVS can start from a very good initial value, which can greatly improve the efficiency and speed of learning. Furthermore, hardware implementation of AVS is much simpler and more effective than in a CNN and for most applications the AVS only requires simple logical calculations. Finally, AVS is based completely on the mechanism of the visual system, so AVS is more reasonable than black-box systems such as neural networks and convolutional neural networks.

3. Results

In order to prove the effectiveness of our proposed mechanism and the AVS based on this mechanism, we randomly generated a large number of different $32\times 32$ pixel images to test. We scanned every pixel of the two-dimensional images with a $3\times 3$ window, used four planar orientation-detective neurons to extract the local planar orientation information at every pixel of the two-dimensional images, and made a judgement of the global planar orientation information based on the local planar orientation information. First, we chose four bars in three different planar orientations to test the proposed mechanism. The first two bars were set at 135 degree angles and had different length-width ratios, whereas the remaining two bars were horizontal and vertical, respectively. In all computer simulations, we scanned with a $3\times 3$ window and the step size was set to 1. The data from each scanning process were transferred to four planar orientation-detective neurons during the scanning procedure, and we counted when the corresponding neurons fired. The experimental results are shown in Figure 6, Figure 7, Figure 8 and Figure 9. We set the fired neurons to 1 and the unfired neurons to 0. We used a simple function diagram to represent the output process of the four neurons and the types of detective neurons were labeled in the graph. Finally, we recorded the total number of activations and picked up their maximum. The activations of the four kinds of neurons are represented in the graph. The horizontal coordinate indicates the serial number of the corresponding scanning window, and the vertical coordinate indicates whether the neurons were activated or not. The number of activations is given and the orange box indicates the maximum value, which is our final judgment for global orientation. From Figure 6, we can see that the neurons for angles of $0^{\circ }$, ${45}^{\circ }$, ${90}^{\circ }$ and ${135}^{\circ }$ fired 38, 0, 39 and 74 times, respectively. Thus, the detection of ${135}^{\circ }$ of planar orientation can be inferred.

Similarly, ${135}^{\circ }$, horizontal ($0^{\circ }$) and vertical (${90}^{\circ }$) orientations can also be inferred from Figure 7, Figure 8 and Figure 9. Finally, we selected seven standard rectangles in ${90}^{\circ }$ with different length–width ratios, and then used a bar chart to show the activation rates of each rectangle. The bar charts of experiments are presented in Figure 10, where the x-axis denotes the length–width ratios and y-axis represents the activation rates of four local planar orientation-detective neurons, with the length of the bar fixed at 30 pixels. According to this experiment, we found that the activation rate decreased with the decrease in the length–width ratios. The closer the rectangle was to a square, the more difficult it was to identify the planar orientation of the rectangle. For a square, with the length–width ratio of $1:1$, the firing rates of the neuron for $0^{\circ }$ and the neuron for ${90}^{\circ }$ were same because even humans cannot distinguish whether a square is at $0^{\circ }$ or ${90}^{\circ }$. When the length-width became to $1:2$, the detective neuron for the $0^{\circ }$ planar orientation fired the most, thus indicating a $0^{\circ }$ orientation detection. This proves that our proposed mechanism is very close to the orientation detection mechanism of the human visual system. Based on all of these computer experiments, we found that our proposed mechanism can accurately detect the planar orientations of objects with different positions, length–width ratios and sizes. Therefore, we can conclude that our proposed mechanism is highly accurate in detecting objects in a specific planar orientation, which also suggests that our hypothesis about local planar orientation-detective neurons and the global planar orientation inference system is possibly correct.

In order to compare the planar orientation detection performance of the AVS with other methods, we selected a CNN because these networks are widely applied with great success in the detection, segmentation and recognition of objects in images. The CNN used in the experiments comprises seven layers: (1) a convolutional layer with 30 feature maps connected to a $3\times 3$ neighborhood in the input; (2) a ReLu layer; (3) a Pooling layer with $2\times 2$ maximum pooling; (4) an affine layer with a full net from 1024 to 720; (5) a ReLu layer; (6) an affine layer with a full net from 720 to 4 and (7) a Softmax layer. The inputs were $32\times 32$ pixel images. The data used to train and test the system were 15,000 and 5000, respectively. The sizes of objects were from 3 pixels to 100 pixels. Learning was performed with backpropagation using the Adam optimizer. All computer experiments were conducted using a PC with an AMD Ryzen 5 3500 6-Core processor and the computational time was measured. The computational times of the CNN for learning and testing were 19.35 s (30epochs) and 1.91 s, respectively. On the contrary, because AVS does not need learning, its computational times required for learning and testing were only 0 s and 0.66 s, respectively, showing that it is faster than the CNN. The identification accuracy of both the CNN and AVS is summarized in

Table 1. Comparison of identification accuracy between CNN and AVS.

Noises	0 Noise	1 Noise	5 Noises	10 Noises	25 Noises	50 Noises	100 Noises	150 Noises
CNN	99.85%	97.89%	59.28%	51.42%	38.04%	35.42%	30.68%	29.38%
AVS	100%	100%	100%	100%	100%	100%	100%	100%

. As expected, the CNN learned the planar orientation detection very well and reached 99.85% identification accuracy. The CNN did not performs as well as the AVS, which showed 100% accuracy. In order to compare the anti-noise ability of both AVS and the CNN, we added noise to the non-object area randomly and these noises were independent of each other and not connected to the object. Figure 11 shows an example image with 0, 1, 5, 10, 25, 50, 100 and 150 noises. Then, we used both AVS and CNN to detect the planar orientations of these object images with noise. Table 1 shows the identification accuracy of both systems, with data presented in Table 1. However, we can see that even if only one source of noise (a pixel) is added, CNN’s identification accuracy drops from 99.85% to 97.89% immediately. As the number of noises increased to 150, CNN’s identification accuracy dropped dramatically, even lower than 30%. In contrast, AVS was always able to maintain 100% identification accuracy, showing superior noise resistance.

4. Conclusions

This paper describes a mechanism for detecting global planar orientation by introducing local planar orientation-detective neurons to compute local planar orientation, and a scheme to judge global planar orientation based on local planar orientation information. That is to say that within a local receptive field, the local planar orientation-detective neurons can extract elementary visual features such as planar orientation. These features are then combined by the subsequent layers in order to detect higher-order features, for example, the global planar orientation. The proposed mechanism has many desirable properties that would be useful in any visual perception system and that seem to be an important part of the human visual system. The mechanism can be used as a framework for understanding many other basic phenomena in visual perception, including the perception of motion direction; the perception of motion speed; and the perception of binocular vision. Furthermore, the mechanism provides a functional architecture for visual computation in the primary visual cortex and provides unprecedented insights into how visual inputs are fragmented and reassembled at different stages of the visual system and how functions are divided across different element of the visual circuit. This mechanism of the primary visual cortex as a sensory system might also help us to understand how other sensory systems, such as olfaction, taste and touch, are encoded at the level of cortical circuits. Although the mechanism is based upon a highly simplified model and ignores most of what is known about the detailed functioning of the visual system and our brain, it does provide a mechanism to explain many known neurobiological visual phenomena in a quantitative manner, and might lead neuroanatomists and neurophysiologists to reexamine their observations, looking for corresponding structures and functions. Conversely, advances in the biological sciences might also lead to a modified and elaborated mechanism.

Based on this mechanism, we developed an artificial visual system (AVS). In order to compare the performance of the AVS and the CNN, we applied the AVS without learning and the CNN with learning to planar orientation detection and found that the AVS performed much better than the CNN in terms of accuracy and noise resistance, as well as in all other aspects. The AVS can be easily applied to other visual perceptions, such as the perception of motion direction, the perception of motion speed and the perception of binocular vision, and even to other sensory systems, such as olfaction, taste and touch. Therefore, we believe that the AVS is very likely to replace the CNN in the near future.

The most important novelty of this paper is that (1) we first proposed a mechanism to explain planar orientation detection in a quantitative manner and verified it through computer simulations successfully; (2) we predicted that local planar orientation-detective neurons might have a $\gamma$ cell-like morphological shape and (3) based the proposed mechanism of planar orientation detection, we developed a generalized artificial visual system (AVS) and showed its superiority to a traditional CNN. In this paper, for the sake of simplicity, we only discussed four planar orientation detection problems, but they are potentially expandable to more planar orientation detection problems as long as we simply increase the sizes of the local receptive field. Similarly, although we have limited our discussions to binary images, we can easily extend the mechanism to grayscale and color images by introducing horizontal cells. Although our mechanism did explain most biological experimental results, the proposal has not yet been directly verified by biological experiments. In future works, our model needs to be confirmed through biological experiments, which may lead to a modified and elaborated model.