Adaptive Neural Networks Accounted for by Five Instances of “Respondent-Based” Conditioning

Neural networks may be made faster and more efficient by reducing the amount of memory and computation used. In this paper, a new type of neural network, called an Adaptive Neural Network, is introduced. The proposed neural network is comprised of 5 unique pairings of events. Each pairing is a module, and the modules are connected within a single neural network. The pairings are simulations of respondent conditioning. The simulations do not necessarily represent conditioning in actual organisms. In the theory presented here, the pairings in respondent conditioning become aggregated together to form a basis for operant conditioning. The specific pairings are as follows. The first pairing is between the reinforcer and the neural stimulus that elicits the behavior. This pairing strengthens and makes salient that eliciting neural stimulus. The second pairing is that of the now salient neural stimulus with the external environmental stimulus that precedes the operant behavior. The third is the pairing of the environmental stimulus event with the reinforcing stimulus. The fourth is the pairing of the stimulus elicited by the drive with the reinforcement event, changing the strength of the reinforcer. The fifth pairing is that, after repeated exposure, the external environmental stimulus is paired with the drive stimulus. This drive stimulus is generated by an intensifying drive. Within each module, a “0” means no occurrence of a Pairing A of Stimulus A and a “1” means an occurrence of a Pairing A of Stimulus A. Similarly, a “0” means no occurrence of a Pairing B and a “1” means an occurrence of a B, and so on for all 5 pairings. To obtain an output, one multiplies the values of Pairings A through E. In one trial or instance, all 5 pairings will occur. The results of the multiplications are then accumulated and divided by the number of instances. The use of these simple respondent pairings as a basis for neural networks reduces errors. Examples of problems that may be addressable by such networks are included. This paper proposes a new way of creating neural networks, called Adaptive Neural Networks. The framework on which these neural networks are based is the Model of Hierarchical Complexity (MHC) (Commons, Gane-McCalla, Barker, & Li, 2014; Commons & Pekker, 2008), a behavioral-developmental and evolutionary theory. This section gives a brief outline of what is being proposed and why it is being proposed. It also lays out the structure of the overall paper.

were three such steps.In this paper, we propose a neural network composed of five pairings.These are similar to the pairings seen in the earlier respondent conditioning paper, but two additional pairings have been added.This Adaptive Neural Network would be faster than the current networks based on operant conditioning, as well as require less memory and use fewer computational resources.While this idea originated from work suggesting that respondent conditioning formed the basis for operant conditioning, the proposal here is not meant as a simulation of any animal behavior.Instead, it is a proposal of how to construct, in the abstract, learning that ultimately is responsive to consequences.
The paper has the following sections.First, in order to understand the basis for such a proposal, the paper introduces the Model of Hierarchical Complexity (Commons et al., 2014;Commons & Pekker, 2008), which is a model of how more complex actions are composed from combinations of simpler actions.The model has been shown to explain developmental differences (i.e., differences between tasks successfully solved by individuals at different ages).It has also been used to explain differences in the complexity of tasks solved by different species of animals (Harrigan & Commons, 2014).
Once the overall model is described, then the proposed early stages of development, including respondent and operant conditioning, will be described.This will provide a context for the proposal for Adaptive Neural Networks.Next, the specific pairings that are proposed for the Adaptive Neural Networks will be described in detail and examples will be given.Following that section, there will be a brief introduction to some of the ideas underlying neural networks, and then a description of the specific form of neural network introduced here.At the end of the paper, we will include a brief summary of the advantages of both the Model and of the Adaptive Neural Networks, as well as implications and likely next steps suggested in this area.

The Model of Hierarchical Complexity (MHC)
The structure of the proposed model for Adaptive Neural Networks is derived from the Model of Hierarchical Complexity.For this reason, the model will be described in some detail.The Model of Hierarchical Complexity (MHC) is a model used for the sequencing of the difficulty of tasks.In order to understand the model, it is first useful to think of the environments in which animals live as consisting of a series of tasks.The tasks generally are organized into domains, such as tasks that are food related, those that are related to producing the next generation, those related to maintaining one's own life, those related to successfully rearing one's young, and so on.Not all domains will be relevant for all animals.One other important aspect of tasks is that there is almost always a sequence of tasks within a domain.That is, one can observe that initially young organisms either do not carry out certain tasks at all or that they carry them out imperfectly.For many animals, one can see development over time in the difficulty of tasks that an organism undertakes.
The question that the Model of Hierarchical Complexity addresses is how to capture this notion of the difficulty or the complexity of tasks.There are two forms of complexity.Horizontal complexity means adding more units, without fundamentally changing the nature of the problem.An example of horizontal complexity is that, in humans, children first learn to add together single digits.While they gradually add larger and larger numbers, they continue to engage in addition but simply execute it for larger numbers.
The Model of Hierarchical Complexity defines a different kind of complexity, as follows.An action is more hierarchically complex when (1) it is defined in terms of at least two task actions from the next lower (adjacent) order of hierarchical complexity, (2) it organizes these less complex actions, and (3) the ordering of the lower task actions has to be carried out in a non-arbitrary way.That is, actions cannot just be chained together in any way.
The higher the order of complexity, the more difficult a task gets.The upside down tree-like structure of the model is shown in Figure 1 below.
Figure 1.Lower order (simpler) tasks, shown at the bottom, must be completed in order to proceed to the higher order (more complex) tasks.Order n+1 is more difficult a task than Order n and less difficult a task than Order n+2.
A useful intuitive example for most readers is seen in the difference between calculating a problem using simple addition or simple multiplication versus carrying out a calculation that involves the distributive property.Children in the early elementary years learn to carry out simple, single arithmetic operations, such as addition and multiplication.So, they could easily solve a problem such as "3 + 4 = ?".Only after they have practiced both addition and multiplication over a reasonably long period of time do they begin to correctly address the distributivity problem (e.g., "2 * (3 + 4)").They do so by coordinating together the two lower order actions of addition and multiplication: 2 (3 + 4) = (2*3) + (2*4)".The distributivity problem is at a higher Order of Complexity because it combines two actions that are from the next lower order.
One additional characteristic of this model is that when an animal completes a task at a particular Order of Hierarchical Complexity, the animal's performance on that task is said to be at a stage that has the same name and number as the order of complexity of the task.The main purpose of having the two terms (Order of Complexity of the task, and Stage of Performance on a task) is that this allows for a clear distinction between the characteristics of the task and the performance on the task.
In this model, behavioral tasks and subtasks are organized according to their hierarchical complexity using an ordinal, equally-spaced one-dimensional scale called the Order of Hierarchical Complexity.The scale is derived from notions proposed by the theory of measurement (Krantz, Luce, Suppes, & Tversky, 1971;Luce & Tukey, 1964).Thus, we consider this to be a mathematical behavioral-developmental theory.It is also evolutionary because it accounts for patterns of task completions of all species of animals, past and current.
Another important influence on the order of hierarchical complexity scale was the work of Piaget and colleagues (e.g., for a comprehensive review, see Hunt, 1969; see also Inhelder & Piaget, 1958;Piaget, 1928Piaget, , 1930Piaget, , 1952Piaget, , 1960)).Piaget and colleagues proposed a notion of stages of development.They published considerable amounts of data over many years showing that children at different ages showed replicable differences in how they approached problem solving.They referred to these differences as stages of development.However, they attributed stage to developments in hypothetical mental structures.The problem was that they never manipulated the tasks used to assess stage and so could never adequately explain performance.The work also relied overly much on verbal justifications, which made it difficult to apply to all animals, including humans.
Other, later conceptions of stages of development, such as that of Fischer (1980) were also influential.Fischer's model was limited to explaining human behavior.Stage ideas have only sometimes been applied to nonhuman animals.One example is work by Parker and McKinney (1999), who applied a few of Piaget's stages to apes.It is asserted here that any theory of the development of intelligence must be applicable to all animals, including humans.
Using this model, 17 Orders of Complexity have been described.The Orders of Complexity have been shown to capture important aspects of task completion across two types of studies.In one type, researchers have classified behaviors of nonhuman or human animals, and then examined the relationship of the Stage of Complexity of a performance to another real world behavior.For example, Miller et al. (2015) found a significant relationship between the hierarchical complexity of a street peddler's pricing strategy and their reported income from street peddling, r = .71,n = 45, p < 0.01.In the second type of study, a series of tasks is constructed, based on the orders of complexity, and these are tested to see if performance on the tasks confirms the predicted order.This uses Rasch (1980) analysis to measure to extent to which performances conform to the predicted sequence.In studies thus far with humans, it has been shown that the orders predict performance with r's ranging from .88 to .98 across a variety of constructed tasks (e.g., Giri, Commons, & Harrigan, 2014).
In order to understand the Model of Hierarchical Complexity in somewhat more depth and, in particular, to understand the implications of its use for examining and comparing the behaviors of different species of animals, as well as its applicability to neural network models, we briefly include some implications of its use in the next section.

Some Implications of this Model
As discussed above, irrespective of context or domain, each order lays the foundation for the next higher order.The other important characteristic of the model is that the method classifies performances only, not animals themselves, and each performance is tied to a specific task.
These two characteristics of the model result in some important predictions.First, the model does not classify an animal as being at a stage.Some tasks that any animal (nonhuman or human) completes may be relatively simple.For instance, a simple reflexive sneeze tends to effectively clear irritants from a nose.Other tasks may be more complex, such as learning how to use a stone to crack open a nut.This kind of heterogeneity across tasks is predicted by the model.
Another prediction that follows from the two characteristics mentioned above is that because of what tasks they successfully complete, there are differences between species.For example, some animals may obtain food by bumping into it in the course of locomotion and then may have a pre-wired mechanism for automatically ingesting it if it meets certain criteria.Others may hunt specific prey, either alone or along with conspecifics.These two ways of solving the task of obtaining food involve actions that are at different orders of complexity, although both types result in the same end (see Harrigan & Commons, 2014, for additional examples).
A third set of implications is that, within species, there can also be differences between different individuals.This heterogeneity is likely to exist in the orders of tasks successfully addressed, and this is true for all species.These may be due to differential learning histories as well as other factors.
Finally, it is important to understand what we mean when we use the term task action.A task action is any action that addresses a certain task.Some task actions correctly address the task and some do not, as is explained next.Both the Orders of Complexity and the Stages of Performance are generally presented in terms of the particular lower order actions that are coordinated together to reach the next higher stage of performance.It is important to recognize, however, that the process of transition from one stage to another is a dynamic one, in which a number of tasks, sometimes called subtasks, will be engaged in as an organism transitions from a task at one order of complexity to a task at the next order.Siegler (1996), among others, has described subtasks engaged in by children as they transition from an easier to a more complex task.Even when an animal has correctly performed a new task at a given order of complexity, actions at previous orders are still available and can be seen under a variety of circumstances.As a result, there will be many behavioral performances by animals, which may or may not directly fit the orders as described.In particular, some nonhuman and human animals may be found to be performing a particular task somewhere within the transition between two of the orders described.
In sum, the Model of Hierarchical Complexity provides a way to scale and compare the difficulty of tasks.Task difficulty has been shown to be related to development within a species.The model provides a methodology for comparing task completions across species, especially with tasks that vary in difficulty.It also provides a methodology for constructing sequences of tasks.These features of the model thus also provide templates for constructing models of problem solving.As we will show later in the paper, one of the applications is to a variety of Artificial Intelligence models.Because the application to be described here is based on Orders 1, 2, and 3 of the model, we will start by describing the first five orders.

The Lowest Orders of Hierarchical Complexity
Brief descriptions of the first five orders are shown in Table 1.Note that Order 0, Calculatory, is included at least partly because the Orders of Complexity do not apply just to animals.They can and should also apply to any entity that carries out a task, even one that is completely preprogrammed and has no variability, such as a computer program.The difference between Calculatory and Automatic is that even though behaviors seen in animals completing tasks at the automatic order are preprogrammed there is also the possibility of some variability in the responses.They are not always carried out in the same exact fashion, the way that they would be in a computer program.This variability in responses would be useful in terms of an organism's likelihood of successfully adapting in the face of changes in the expected environment.
In the next section, the paper will expand on the information given in Table 1, in particular for Orders 1, 2, and 3, because those will be the orders that are most relevant to the Adaptive Neural Networks proposal.The paper will provide some animal examples.These examples will help to make clear what kinds of tasks characterize an Order of Complexity, as well as how more hierarchically complex tasks are combinations of lower order tasks.Example: When infant babbling is followed by vocalizing and smiling from adult, infant babbles more.
Example: Animals from a variety of species learn discriminations of concepts, such as same/different.

The First Three Behavioral-Developmental Orders
Automatic Order 1 At the Automatic Order 1, a single action occurs as an innate biological response to a single environmental stimulus.This may be a reflex or a tropism or other related action.There are many animals that successfully perform tasks at this Order of Complexity.The examples provided here are primarily from unicellular organisms.It is important to keep in mind, however, that this discussion is not characterizing any particular organism or type of organism as being at a stage.Instead, the discussion focuses primarily on the characteristics of an Order 1 task.It does include an example of an animal performing an action that completes that task.
In the case of Automatic Order 1, a single response is elicited by a single environmental stimulus.In a particular organism, there is no variability in the response seen to that stimulus.An example of Order 1 response is a form of taxis seen in an experiment by Armus and Montgomery (2001).They showed that paramecia approach areas in which a mild shock was delivered.In that case, the electrical field created by the shock is the environmental event, and the approach behavior is the response.
There is behavioral change at this order because there are processes such as stimulus generalization, habituation, and sensitization.However, the eliciting stimulus is not coordinated with other environmental stimuli as in respondent or classical conditioning (Commons & Giri, 2016).

Sensory or Motor Order 2
Respondent conditioning at Order 2 of hierarchical complexity results from the coordination of two stimulus response pairs.One of the stimulus response pairs is an unconditioned reflex.For example, an unconditioned stimulus (US), such as an air puff, elicits an unconditioned response (UR), an eye blink.The second pair consists of an event that automatically elicits orienting or attention.This might be an auditory or visual stimulus.When the auditory or visual stimulus is paired with the US (air puff), because it already elicits attention, it can become a conditioned stimulus (CS).Over time and enough pairings, it comes to elicit the eye blink (conditioned response or CR).This example comes from the work of Thompson and colleagues on classical conditioning of the eye blink reflex (e.g., Thompson et al., 1998;Weeks et al., 2007).
In addition to showing that simple auditory and visual stimuli, when paired with an unconditioned stimulus, such as an air puff, would then elicit the previously unconditioned eye-blink response, Weeks et al. ( 2007) also have contributed elegant work showing the possible brain circuits underlying this kind of classical conditioning.Note that the pairing may either occur in a naturalistic environment or could be carried out by an experimenter.It is also important to mention that the example above involves a reflex.It is likely that other types of Order 1 behaviors could be part of a respondent conditioning Order 2 task.For example, Armus, Montgomery and Gurney (2006) reported that paramecia differentially swim to areas that are paired with a light that had been previously paired with shock.While most previous work on paramecium learning has found there to be some confounding factor that accounts for the apparent learning, thus far no such criticism has been aimed at the Armus et al. (2006) study (see Mingee, 2013).Absent at least some basic neural circuitry, it is hard to see what that mechanism might be.
In sum, Sensory or Motor Order 2 actions coordinate two or more stimulusresponse pairs from the lower Automatic Order 1.The response that is paired with the stimulus can be any of the Order 1 Automatic behaviors.

Circular Sensory Motor Order 3
The definition from the Model of Hierarchical Complexity states that operant conditioning (Order 3) is the product of the nonarbitrary coordination of two or more respondent conditioning (Order 2) pairings (Commons & Giri, 2016).This section will first briefly describe the traditional view of operant conditioning and give two examples.A more detailed description of the specific classical conditioning pairings that constitute operant conditioning will be laid out in the next section.
Operant conditioning has been described as resulting from a sequence of three events, specifically, an observable behavior that occurs in a stimulus situation and is followed by either a reinforcing or a punishing consequence.This kind of conditioning can be seen in at least some invertebrates as well as many other kinds of animals.The studies of invertebrates have also often connected the appearance of operant conditioning to some form of simple nervous system.Drosophila melanogaster (fruit flies) have been operantly conditioned in a laboratory environment (Brembs & Heisenberg, 2000).Aplysia californica (California sea hare) have also been operantly conditioned by pairing electric stimulation in the anterior branch of the esophageal nerve (En2) with biting behavior (Lechner, Baxter & Byrne, 2000).
In the current paper, a different conception of operant conditioning is proposed, which consists of a coordination of five pairings of respondent conditioning instances.These five are based on pairings of events that are the same or similar to the ones involved in respondent conditioning.Thus, the clear implication is that operant conditioning is based on respondent conditioning.
There are several reasons for making such a proposal.As argued in more detail in Commons and Giri (2016), the accounts of operant learning are not as well worked out as those of respondent conditioning.They also contain several problems.One problem is that it is not clear why a behavior that becomes an operant starts occurring in the first place.Commons and Giri (2016) argued that the stimulus situation prior to the behavior has been neglected as an important factor in causing that behavior.A second difficulty is that cause  effect chains typically run from left to right.Most accounts of operant conditioning nevertheless assert that the cause of the behavior becoming an operant (the reinforcer) actually happens after the behavior.Finally, if one considers how a new learning mechanism might evolve, it would seem most likely for any new mechanism to be based on existing, older mechanisms.The evolutionary record suggests that very simple animals, in which it is likely that either only Order 1 or at most Order 2 behaviors were present, were likely the first animals.The most straightforward way for later learning mechanisms-such as operant conditioning-to develop is partly basing on earlier learning mechanisms.Presumably, initially some small incremental changes might have occurred in respondent conditioning.Animals who exhibited additional pairings might have gained an advantage in terms of their obtaining of resources and eventual survival.
As a result of these arguments, Commons and Giri (2016) argued that a better explanation of operant conditioning was that it consists of three instances of respondent conditioning that are coordinated together in a nonarbitrary fashion.As can be seen by the reader, the idea of this coordination is based on the Model of Hierarchical Complexity.That is, it requires that the higher order action of operant conditioning be the result of nonarbitrary coordination of at least two respondent conditioning actions.In further elaborating on this idea in this paper, we have found that five respondent-based pairings are necessary to completely account for operant conditioning.

Five Steps in Operant Conditioning
The following sections will explain the five respondent-conditioning steps that make up operant conditioning.The steps will be related to the five questions associated with each of these steps.Table 2 summarizes in a short, symbolic form the relationships described in the text.Note that in Table 2 the specific situation that is included is one in which there is positive reinforcement.Also note that the step numbers give the ordering in which the steps occur.One assumption that underlies this conception is that all behaviors are caused by stimuli, either external or internal.Even if a free operant behavior appears to be random, it is posited that there is an internal stimulus that elicits it.This assumption is necessary because the original cause of the behavior has to be something that occurs before it, not afterwards.Because it is an internal event, it is not always directly observable.Evoked-potential studies often find this stimulus occurring 250 to 500 ms before the operant response (Sur & Sinha, 2009).In this work, we refer to this internal stimulus or internal neural event as the s rb .It is equivalent to a US/CS, albeit an internal one.
In the next section, the five steps are described in detail and an example is given in support of each description.

Step 1: What Is the Value of Doing It? Or, Drive-Determining Consequence Value
In Step 1 from respondent conditioning, the activation of the drive stimulus (S Drive ) gives value to the consequence S Consequence .That is, the pairing of a drive stimulus with a consequence changes the value of the consequence simultaneously.The result is that the consequence, S consequence , changes to S R+ , a reinforcing consequence.This can also be referred to as the US/S R+ .
One can see the importance of the drive stimulus in the common practice of maintaining animals at 80% of their free-feeding weight when food is being used as a reinforcer.This is well understood to increase the value of the food consequence.This increases the drive level, which has the effect of increasing the value of the consequence.This makes learning more likely.
Step 2: What to do?Or, Operant Strengthening In Step 2 from respondent conditioning, the internal neural event (s rb ) that elicits the operant response is paired with the S R+ .The association of the neural event (s rb ) with the US/S R+ makes the internal neural event more salient and thereby helps to strengthen the operant response.
An example that illustrates this can be seen in the work of Lechner, Baxter, and Byrne (2000) with Aplysia californica.In a classical conditioning procedure, tactile stimulation of the lips of Aplysia with a paintbrush was paired with seaweed.Conditioning was tested simply by comparing the change in the number of bites that an animal made to the seaweed in the paired condition as opposed to those in the unpaired condition.While this was not set up as an operant learning situation, the animals were allowed to ingest the seaweed for up to 60 s following the beginning of its presentation.This, in effect, served as an event that could reinforce earlier events.Furthermore, the researchers found that a key component of the conditioning that occurred was having an intact esophageal nerve.When other relevant neurons were tested, there was no such effect.The findings were that the tendency to respond to the US was strengthened by the pairing of the internal neural event and the food (S R+ ).

Step 3: When to do it? Or, Stimulus Control
The stimulus control process is the pairing of the now more salient neural stimulus s rb (along with the operant response) with the environment event S Environment .This is a "when" pairing because the cue or cues in the environment elicit the neural stimulus, s rb , determining when it occurs.When the relevant cues are not here, the neural stimulus does not occur.In operant terms, this pairing changes the environmental event into an SD (discriminative stimulus).Both S Environment and s rb have to be salient in order for learning to take place (Rescorla & Wagner, 1972).The internal stimulus becomes controlled by the occurrence of the environmental stimulus no matter what the time difference is.
An experiment by Andrew and Savage (2000) shows this phenomenon in Lymnaea (pond snails).In one of the sub-experiments, snails were placed in a glass gutter with either a visible black or a white panel 30 cm into the tube.The black panel was judged to be discriminable, while the white panel was not.In reinforcement conditions, these potential conditioned stimuli were paired with sucrose (or water in the nonconditioning trials).Snails were observed to rasp (which is the way a snail would ingest food) in anticipation of the possible delivery of sucrose much sooner for trials with black panels than those with white panels.Thus, snails showed conditioned feeding in the presence of a visual (environmental) event.
Step 4: Why to do it?Or, Establishing an Incentive In Step 4 from respondent conditioning, the environmental stimulus S Environment is paired with the S R+ , making the S Environment more salient and valued.Pairing the environmental stimulus with the reinforcing stimulus establishes the environmental stimulus S Environment as an incentive (see Killeen, 1982).The incentive value means that there is an increase in the salience and value of the representation of a reinforcement rate relative to the representation of other behaviors that are not associated with reinforcement.
An example is seen in Octopus vulgaris (Schiller, 1949).The organism is placed in a maze.There are two inverted cans.One of the cans had covered crab bait; the other can was without crab bait.Thus, the environmental stimulus S, the "visible" bait can, was paired with crab bait.A partition wall had to be circumvented to reach the baited can.Octopus vulgaris learned to make a turn toward the proper side if the bait was visible at all times as it viewed it as an incentive.
Step 5: Where to do it?Or, Environment Paired with the Drive Stimulus In Step 5 from respondent conditioning, the environmental stimulus S Environment gets paired with the drive stimulus, S Drive .After multiple trials of this type of pairing, the properties of the environment or a similar environment are paired with drive stimulus S Drive .The organism will then react to an environment where there is a drive associated with it.
This can be illustrated by experiments involving the effects of viewing food images that are high in fat or carbohydrates (Harrar, Toepel, Murray, & Spence, 2011).When shown such images, human participants responded more rapidly the higher in fat or carbohydrates a food was perceived to be.The authors interpreted this as showing that because such food items have a higher "incentive" value, seeing them may produce a kind of overall alerting effect.In this case, "higher incentive value" refers to the drive stimulus.Spence, Okajima, Cheok, Petit, & Michel (2016), in an extensive review, summarized multiple ways in which exposure to food images in humans' everyday environments is associated with a greater tendency to eat and, therefore, with being overweight.
The description of the five respondent conditioning-based steps above represents a possible model for the respondent pairing steps that underlie operant conditioning.Recall that it is not being said that this is necessarily how operant conditioning takes place in live organisms.What this model does is take all the events involved in operant conditioning and interrelates them using five instances based on the existing and known mechanism of respondent conditioning.This is possibly useful in at least two ways.For one thing, research on automatic, respondent, and operant behaviors can empirically test various aspects of this model with animals.A second application of these ideas is to translate the events and their inter-relationships into a proposed neural-network model.Having the steps clearly delineated in terms of the type of events that each represents is necessary for such modeling.In the next section, we will first describe and define what a neural network is.Then, we will show what the proposed neural network for the five steps looks like and how it would operate.

Neural Networks
Neural networks are one possible implementation within the larger field of what is called Artificial Intelligence intelligence (AI).The main goal of AI is to represent information processing and problem solving actions seen in animals using either hard wired machines or computer programs (Goertzel & Pennachin, 2007).As argued in Leite (2018), AI can be used as a model for testing different models of how human and nonhuman animals solve tasks (see also Cassimatis, 2012).
A neural network is a particular kind of artificial information processing system that simulates the nervous system.It is thought to be closer to the way that the brain actually works.A particular neural network may consist of multiple units or nodes.The nodes are equivalent to artificial neurons or groups of neurons.These nodes are interconnected with each other to form a network (Guresen & Kayakutlu, 2011).The main objective of a neural network is to find associations and patterns with the input and outputs through pattern recognition.The associations are formed by training using feedback from the output to the input for correct indications of the pattern.Figures 3a and 3b show different visualizations of a neural network.
A hypothetical three-layer neural network is shown in Figure 3a.We will use this simple diagram to illustrate some of the possibilities of neural network models.Within this illustration, we will use a tangible example of sensory inputs being converted into the output of recognizing an appropriate prey animal.This example was simply made up here to illustrate some of the features of these models.One starts to read such a model often at the bottom.So, at the bottom of this model, inputs (represented as black arrows) are shown entering or being registered by each of the red circles.These inputs might be sensory information, for example, sounds, smells, and visual inputs from a stimulus.As shown by the pink arrows going from the bottom to the middle row of nodes, each node at the bottom is shown to be feeding the information received to more than one node at the second level.In effect, this means that each of those three nodes at the second level would combine the same information, although if they are postulated to be in different parts of a brain, the resulting output might be different.For example, perhaps one of the middle circles is a kind of visual processing node, and another is an odor processing node.
The top-most node receives input from each of those middle nodes.This leads to a further combination of the information.A resulting output might be the recognition of a prey animal.Most importantly, depending upon how a researcher diagrammed the model and the processes that they were interested in, they could construct these models in different ways.For example, the model in Figure 3a contains three layers.It is possible to construct models with either fewer or more layers.The circles, or the nodes themselves, could be programmed to carry out different actions or operations.Literally then, this provides a way to translate a particular information processing or problem solving process into a kind of structure to test the extent to which the proposed neural network model solves that same particular problem in the same way that animals are thought to solve that problem.The solving of the problem process is often done through the use of a computer program.Doing this clearly forces the precise specification of the possible processes involved.

Subfigure 3a
Subfigure 3b Step 4 (why to do it), and Step 5 (where to do it).Note that two possible inputs are shown.This represents the fact that animals are almost always in environments with multiple inputs.As mentioned in the Figure 4 caption, it is assumed that these inputs or environments will usually have different rates of reinforcement.Also note that Step 3, the where step, pairs the internal event with the environment.A feedback loop is shown here to represent this idea.Finally, note that the inputs from each step are accumulated and produce the output at the top.
It is proposed that the first layer of a respondently-based neural network (see Figure 4) follows an all or nothing method, in which "0" means no occurrence of a correct pairing at that step and "1" means an occurrence of a correct pairing at that step.In each case, either a 0 or a 1 representing the output is multiplied.When effective pairing has occurred, the value resulting from that step is 1.
Once these initial outputs are obtained, they are fed into a second layer, which cumulates the values from the 5 steps below.To get the relative value of the conditioning, this second layer sums the total number of successful instances, then divides it by the sum of successful and unsuccessful instances.This yields weights reflecting the strength of response and amount of stimulus control.Operant conditioning does not happen unless all the steps are executed.In that case, it is called a complete instance.There are failed instances, in which not all steps are effectively executed.
The simplicity of the pairing, resulting in outcome values of either 0 or 1, which can then be multiplied, will increase processing speed drastically since it uses a minimum amount of memory.This is in contrast to neural networks that are often currently written in higher order languages, such as R, that execute more slowly and use more memory.

Discussion and Conclusions
This paper introduces a proposed mechanism for operant-conditioning-like neural networks, based on five steps of respondent conditioning.The proposed mechanism is an application of the Model of Hierarchical Complexity.This model has been shown to predict changes in development within a species (e.g., Giri et al., 2014, Commons & Harrigan, 2014,).The order of complexity of a task completed has also been shown to predict real world outcomes, such as earnings in street peddlers (e.g., Miller et al., 2015).Most importantly in this context, the model also provides a framework for a more precise understanding of differences in successful task completions across different animals (e.g., Harrigan & Commons, 2014).
The Model of Hierarchical Complexity categorizes operant conditioning as one order of hierarchical complexity above respondent conditioning.By definition, because higher order tasks are the result of coordinating at least two less hierarchically complex task actions, it was argued that coordinating several instances of respondent conditioning together would be one way in which operant conditioning might have arisen in the course of evolution.This proposal shows how all of the antecedent and consequent events are coordinated together to obtain operant conditioning.Finally, it is shown that the five respondent conditioning steps can also form the basis for a relatively simple neural network that would simulate operant conditioning.
There are several implications and next steps related both to the Model of Hierarchical Complexity and to the proposal for a new kind of neural network.One, also suggested earlier, should be more empirical attempts at categorizing task performances in different animals in terms of the Model of Hierarchical Complexity.This would be done mostly by creating sequences of tasks that would fit the sequence of the orders of hierarchical complexity and testing these sequences.For example, with respect to the series of tasks suggested here, an animal may complete some of the five steps but not all of them.It is also possible that empirical testing of these ideas would show that a different number of steps might be enough for operant conditioning to occur.The purpose here is to propose a structure for testing a variety of possibilities in real-world situations.
As also discussed in the introduction to the model, it is likely that performances that are intermediate between the definitions of the orders of complexity are also possible.Siegler (1996), for example, found that children engaged in a number of subtasks on their way to acquiring a task.An animal's task performances may be at various different points in the development from one Order of Complexity to the next.Another next step based on this proposal is to begin to construct and test a neural network model that includes the five respondent pairings.Performance of such a network on simulations of commonly used empirical situations would be an excellent way to better understand the evolution and comparative performances of animals on these tasks and to better understand the development of problem solving in general (as also proposed by Common's Applied Patent, 2015;Leite, 2018).The five-layered nature of the respondent neural networks was suggested in Leite's dissertation (2018).Compared to other currently available neural networks for operant conditioning tasks, the respondent neural networks are more efficient because they are based on simple addition and multiplication of 0's and 1's.Because of the simplicity of the network, it may also be possible to develop hardware based on these principles, in addition to software.Multiplying 0's and 1's as opposed to having to program in higher-level languages in computation can be directly done in machine language.Once versions of the network are developed, via either hardware or software, then they can help Submitted: May 17 th , 2017 Resubmitted: August 20 th , 2019 Accepted: October 13 th , 2019

Figure 3 .
Figure 3.A simple neural network.On the left (a) is a fully connected three-layer neural network complete with hidden layers and a feedback loop.The right subfigure (b) shows the same neural network represented as a single module.The colored square containing the circle with arrows represents a neural network and its hidden layers.The arrows pointing upward into the module represent the input.The pink arrow shows the output.

Figure 4 .
Figure 4.An example of neural networks represented as five steps of respondent conditioning.What is shown here is the first layer of a single respondently-based neural network.The neural network may only pay attention to one of the sensory inputs that are shown.It will give precedence to the signal or input with the larger relative reinforcement rate.

Table 1
MHC Order Names, Numbers, Definitions, and Examples

Table 2
Five Steps of Respondent Conditioning Note. S = stimulus; R = response; S R+ = positive reinforcement; o = pairing