Quantitative Single-Neuron Modeling: Competition 2009

1. DESCRIPTION How well are single-cell properties reproduced by the present-day neuronal models? Recently, several labs have approached this question by assessing the quality of neuron models with respect to spike timing prediction or characteristic features of the voltage trace. So far, every modeler used his own preferred performance measure on his own data set. The Quantitative Single-Neuron Modeling Competition offers a coherent framework to compare neuronal models and fitting methods. 1.1. Participation Participants can submit their prediction to one or more of the challenges A, B, C or D. Anyone can participate and any type of model is accepted. 1.2. Goal This competition is an opportunity to bridge the gap between experimentalists and model-ers. The Quantitative Single-Neuron Modeling Competition is an invitation to compare your methods and models to those of other people in the field. Past competitions remain available on the INCF website and will serve as benchmarks [ 5, 6]. Prizes are given to outstanding contributions.


Goal
This competition is an opportunity to bridge the gap between experimentalists and modelers.The Quantitative Single-Neuron Modeling Competition is an invitation to compare your methods and models to those of other people in the field.Past competitions remain available on the INCF website and will serve as benchmarks [ 5,6].Prizes are given to outstanding contributions.

Prizes
The prizes are given according to the "Rules and Conditions".
• The INCF 1 Prize ( 10 000 CHF) is given to the participant(s) providing a significant win2 in at least 2 of the four challenges.• The FACETS3 Award (500 CHF) is given to the participant providing a significant win 2  in a single challenge, or shared between all participants providing a shared win 2 in a single challenge.

Important Dates
• Submissions via the website will open on June 25 2009.
• August 25 is the submission deadline.
• September 6-8 is the INCF Congress of Neuroinformatics 4 where results are presented.

RULES AND CONDITIONS
1. Winning conditions.The prizes are given upon the disclosure of the model.Winning participants who fail to provide the source code to the organizers in the time between the submission date and the day before the disclosure of the results will be removed from the competition.Once the results have been verified by the organizers and the details of the winning model/method have been published on the competition website the organizers will communicate with the winners to give the money prize.The INCF Prize (10 000 CHF) is payed as a honorary to the participant(s) providing a significant win in at least 2 of the challenges.The FACETS Award (500 CHF) is payed as a honorary to the participant(s) providing a significant win in a single challenge or shared equally between participants providing a shared win in a single challenge.The participants keep the right to publish their model in journals and conferences.
2. EPFL employees.EPFL employees may take part in the competition as regular participants but they cannot receive the honorary linked to the FACETS Award or the INCF Prize.The EPFL employee who wins either INCF Prize or FACETS Award will receive the certificate associated with his prize, his name will appear in the results, but the money prize will not be awarded that year.
3. Shared INCF prize.It is possible that two participants win the INCF Prize, in this case the money prize is shared equally.
4. Significant win.A performance in one of the challenges that surpasses the second best performance in the same challenge by more than one standard error of the mean is considered a significant win.The value of performance and standard deviation is defined in each challenge "Evaluation Methods".There is maximum one significant win per challenge.There can be a significant win in a single challenge only if it receives at least three submissions.
5. Shared win.All participants in a challenge having performances that fall within one standard error of the mean of the best performance are said to achieve a shared win.In this case, the honorary associated with the prize is shared between the best participant and these others.the participant with the best performance is also a shared win.In this way there is either zero or minium two shared win per challenge.The value of performance and standard deviation is defined in each challenge Evaluation Methods.There can be a significant win in a single challenge only if it receives at least three submissions.
6. Collective Submission.Multiple people can collaborate to make a submission.In which case the first name within the collaborators is the responsible author.The responsible author of the submission is called the participant.
7. Single Submission.A participant can only submit once per challenge.

CHALLENGE A
Predict the spike timing of a regular spiking L5 pyramidal cell responding to in-vivo-like current injection.

Experimental Methods
The experiments were performed by Thomas Berger and Richard Naud in the laboratory of Henry Markram at the EPFL.A 14-day-old Wistar rats was decapitated and its brain was quickly transferred to a slicing chamber filled with iced artificial cerebrospinal fluid (ACSF).300 mm thick slices of the primary somatosensory neocortex were prepared using a HR2 vibratome (Sigmann Elektronik, Heidelberg, Germany).Slices were incubated at 36 The amplifier was connected to a ITC-18 acquisition board (Instrutech Co, Port Washington, NY), which was in turn connected to a PC or Macintosh running a custom written routine under IgorPro (Wavemetrics, Portland, OR).Patch pipettes were pulled with a Flamming/Brown micropipette puller P-97 (Sutter Instruments Co, Novato, CA) and had an initial resistance of < 4M Ω. Pipettes were filled with intracellular solution (ICS) containing (in mM): 110 potassium gluconate, 10 KCl, 4 ATP-Mg, 10 Na-Phosphocreatine, 0 3 Na-GTP,10 Hepes, 30 Mannitol, and 8 Biocytin.Chemicals were provided by SIGMA or MERCK.The liquid junction potential between the ACSF and the ICS was around 12 mV and not corrected for.
The current-clamp stimulus has two parts.The first part is 17.5 seconds of various stimulus waveforms frequently used to calibrate neuron models.It consists of a series of four step current with a duration of 2 seconds and an inter-step rest time of two seconds (one hyperpolarizing and 3 depolarizing steps).The steps are followed by an injection of white noise of two seconds.The white noise injection can be used to remove the artefact introduced by the electrode while recording the voltage [ 1,2].The second part of the stimulus takes the remaining 42.5 seconds and is made of a simulated excitatory and inhibitory spike train.Six spike train were simulated as inhomogeneous poisson processes with predefined intensity as a function of time.The first three were convolved with a monoexponential decay of time constant of 2 ms while the last three spike trains were convolved with a mono-expontential decay of time constant of 10 ms.The six resulting time-series were then combined into a single time-series by weighted sum.The weights were chosen such as to drive the neuron to a firing frequency between 5 and 10 Hz, as seen in experiments in vivo.The time-dependent intensity of the poisson process consisted of a concatenation of blocks of 300-500 ms duration each having a constant intensity chosen randomly between 0 and 50 Hz.Both the amplitude and the duration of the blocks were drawn from a uniform distribution.

Evaluation Methods
Thirteen repetitions of 60s-stimulation protocol specified above were injected while the voltage was measured at the same electrode.The training set consists of the first 38 s and the test set consist of the last 22 s.We provide the injected current waveform in pA for both training and test phases.
For training the participants have access to the voltage trace of each repetition for the first 38 s.For testing, the participants must predict the spike times of the remaining 22 s, in each repetition.The spike time is defined as the time at which the voltage recording is crossing 0 mV.The measure of the performance is based on the Gamma coincidence factor.To evaluate this quantity, we calculate the number of coincidences N coinc between the spikes in the data spike train one repetition at a time (target) and the spike train of the model submitted by a participant.This number is calculated by counting the number of target spikes for which we can find at least one model spike within ± 4 ms.We subtract the expected number of coincidences N coinc that a Poisson spike train with the same average frequency would give, and we divide by the number of spikes in the two spike trains: where N data and N model denote the number of spikes in the data and model spike trains, and α is a factor that normalizes the coincidence factor Γ to a maximum of 1.
For details see references [ 4] and [ 5].Γ = 0 implies that the prediction is not better than chance level.Γ = 1 implies that the prediction by the model is optimal.
For each repetition i, we calculate the Gamma coincidence factor Γ i between the predicted spike train and the observed spike train for the last 22 s.The performance is the mean Γ divided by the intrinsic reliability: where N rep = 13 is the number of repetitions.The intrinsic reliability, R, is the averaged coincidence factor obtained by comparing one observed spike train with the spike train from another repetition: The standard error of the mean is computed with the bootstrapping method.A large number (here n =10 000) of samples are generated artificially by re-sampling from the original sample S 0 = {Γ 1 /R, Γ 2 /R, ..., Γ Nrep /R}.Each generated sample S j is made of N rep elements chosen randomly from the original sample with replacement.The re-sampled mean, µ j , is the average of the Γ/R within the sample set S j .The standard error of the mean is then the standard deviation of the re-sampled averages: This bootstrapping method is known to infer the standard error or confidence intervals of measures (such as mean, median, variances, ...) even when the original sample is drawn from a non-gaussian distribution.
We say that a participant j achieves a significant win if its performance P j A is one standard deviation above the performance of all other participants (if P (j) A − Err (j)

A > P (i)
A ∀ i = j then j achieves a significant win).When there is no significant win there are multiple participactions who achieve a similar performance.We say that a particpation makes a shared win if its performance is within one standard deviation of the best performance (if P (j) then j and (best) have a shared win).

Submission
Submissions are made through the INCF website.The participant must provide his prediction of spike times for each repetition.The submission consists of a folder containing 13 files, one for each repetition.Each file contains the predicted spike times in milliseconds (ms) since the beginning of the repetition.Spike times need to be stored as columns in ASCII file format.The files are labeled from 1 to 13 and are called 'repXX.txt'(rep1.txt,... rep13.txt).Submissions must comply with these specifications to ensure that the automatic evaluation of the results is successful.

I(t) V(t)
Predict the spike timing of a L5 fast spiking cell responding to in-vivo-like current injection.

Experimental Methods
The experimental methods for challenge A and B are identical.

Evaluation Methods
The evaluation methods are identical to those of challenge A, with the only exception that there are 9 repetitions available in challenge B.

Submission
Submissions are made through the INCF website.The participant must provide his prediction of spike times for each repetition.The submission consists of a folder containing 9 files, one for each repetition.Each file contains the predicted spike times in milliseconds (ms) since the beginning of the repetition.Spike times need to be stored as columns in ASCII file format.The files are labeled from 1 to 9 and are called 'repXX.txt'(rep1.txt,... rep9.txt).Submissions must comply with these specifications to ensure that the automatic evaluation of the results is successful.
Recording electrodes were pulled from thick-walled (0.25 mm) borosilicate glass capillaries and used without further modification (pipette tip resistance 5-10 MΩ for soma and 20-30MΩ for dendrites).Whole-cell voltage recordings were performed at the soma of layer V pyramidal cells.After opening of the cellular membrane a fluorescent dye, Alexa 594 could diffuse in the entire neuron allowing to perform patch clamp recordings on the apical dendrite 600-700 µm from the soma.Both recordings were obtained using Axoclamp Dagan BVC-700A amplifiers (Dagan Corporation).Data was acquired with an ITC-16 board (Instrutech) at 10 kHz driven by routines written in the Igor software (Wavemetrics).
The injection waveform consisted of 6 blocks of 12 seconds.Each block is made of three parts: 1) one second of low-variance colored noise injected only in the soma, 2) one second of low-variance colored noise injected only in the dendritic injection site, 3) ten seconds of high-variance colored noise whose injection site depends on the block: In the first block, the 10-second stimulus is injected only in the dendritic site, the second block delivers a the 10-second stimulus in the soma only, and the four remaining blocks deliver simultaneous injection in the soma and the dendrites.The colored noise was simulated with MATLAB as an Ornstein-Uhlenbeck process with a correlation time of 3 ms.The six blocks make a 72 seconds stimulus that was injected repeatedly without redrawing the colored noise (frozen-noise).Twenty repetitions of the 72-second stimulus were carried out, separated by periods of 2-120 seconds.Out of the twenty repetitions, a set of seven successive repetitions were selected on the basis of high intrinsic reliability (see "Evaluation Methods").

Evaluation Methods
Seven repetitions of a current waveform of 72 seconds were injected in dendritic and somatic injection site while the voltage was measured with the same electrodes.The training set consists of the first 38 s and the test set consists of the last 34 s.We provide the somatic and dendritic current waveform in pA for both training and test phases.For training the participants have access to the voltage trace of each repetition in each recording site for the first 38 s.For testing, the participants must predict the somatic spike times of the remaining 34 s, in each repetition.
where N rep = 7 is the number of repetitions.
The standard error of the mean is computed with the bootstrapping method.A large number (here n =10 000) of samples are generated artificially by re-sampling from the original sample S 0 = {Γ 1 /R, Γ 2 /R, ..., Γ Nrep /R}.Each generated sample S j is made of N rep elements chosen randomly from the original sample with replacement.The re-sampled mean, µ j , is the average of the Γ/R within the sample set S j .The standard error of the mean is then the standard deviation of the re-sampled averages: This bootstrapping method is known to infer the standard error or confidence intervals of measures (such as mean, median, variances, ...) even when the original sample is drawn from a non-gaussian distribution.
We say that a participant j achieves a significant win if its performance P j D is one standard deviation above the performance of all other participants (if P D ∀ i = j then j achieves a significant win).When there is no significant win there are multiple participactions who achieve a similar performance.We say that a particpation makes a shared win if its performance is within one standard deviation of the best performance (if P

Submission
Submissions are made through the INCF website.The participant must provide his prediction of the geniculate spike trains for each repetition.The submission consists of a folder containing 38 files, one for each repetition.Each file contains the predicted spike time in milliseconds (ms) since the beginning of the repetition.Spike times are stored column wise in ASCII file format.The files are labeled from 2 to 76 and are called 'repXX.txt'(rep2.txt,rep4.txt,... , rep76.txt).Submissions must comply with these specifications to ensure that the automatic evaluation of the results is successful.

s
Predict the timing of somatic spikes of a tufted L5 pyramidal cell responding to in-vivo-like current injected in the apical dendrites and the soma simultaneously.
then j and (best) have a shared win).