Description and Validation of a Computerized Behavioral Data Program: “BDataPro”

Abstract

BDataPro is a Microsoft Windows®-based program that allows for real-time data collection of multiple frequency- and duration-based behaviors, summary of behavioral data (in terms of average responses per min, percentage of 10-s intervals, and cumulative responses within 10-s bins), and calculation of reliability coefficients. The current article describes the functionality of the program. BDataPro is freely available for download from the authors’ institution websites.

Appendices

Appendix A

Reliability Calculation Formulas

Frequency Behavior Reliability Calculations

• a. EIA—Exact Interval Agreement

  • 1. Divide sessions into successive 10-s intervals.

  • 2. An agreement is scored for an interval if both observers record the same number of responses (e.g., both recorded zero responses; both recorded three responses).

  • 3. The total agreements and disagreement for a session is then put into the following formula:

    $$ \left(\mathrm{Number}\ \mathrm{of}\ \mathrm{agreements}\kern0.5em /\kern0.5em \left(\mathrm{Number}\ \mathrm{of}\ \mathrm{agreements}\kern0.5em +\kern0.5em \mathrm{Number}\ \mathrm{of}\ \mathrm{disagreements}\right)\right)\kern0.5em \times \kern0.5em 100. $$

• b. PIA—Partial Interval Agreement

  • 1. Divide sessions into successive 10-s intervals.

  • 2. For each interval, the lower number of responses recorded is divided by the higher number. If both observers record zero responses, a ratio of 1.0 is substituted.

  • 3. The ratios are summed and divided by the number intervals in the session and then multiplied by 100.

    $$ \left(\mathrm{Sum}\ \mathrm{of}\ \mathrm{ratios}\kern0.5em /\kern0.5em \mathrm{Number}\ \mathrm{of}\ \mathrm{intervals}\kern0.5em \right)\kern0.5em \times \kern0.5em 100 $$

• c. TIA—Total Interval Agreement

  • 1. Divide sessions into successive 10-s intervals.

  • 2. For each interval, an agreement is scored if both observers record at least one response or if both record zero responses. A disagreement is scored if one observer records at least one response and the other records zero responses.

  • 3. The total agreements and disagreement for a session is then put into the following formula:

    $$ \left(\mathrm{Agreements}\kern0.5em /\kern0.5em \left(\mathrm{Agreements}\kern0.5em +\kern0.5em \mathrm{Disagreements}\right)\right)\kern0.5em \times \kern0.5em 100. $$

• d. OIA—Occurrence Interval Agreement

  • 1. Calculated in the same manner as total agreement except that only intervals in which at least one response was recorded are included (i.e., if both observers recorded zero responses, that interval would be excluded).

    $$ \left(\mathrm{Agreements}\kern0.5em /\kern0.5em \left(\mathrm{Agreements} + \mathrm{Disagreements}\right)\right)\kern0.5em \times \kern0.5em 100 $$

• e. NIA—Non-occurrence Interval Agreement

  • 1. Calculated in the same manner as total agreement except that only intervals in which at least one of the observers recorded zero responses are included (i.e., if both observers recorded at least one response in the interval, that interval would be excluded).

    $$ \left(\mathrm{Agreements}\kern0.5em /\kern0.5em \left(\mathrm{Agreements} + \mathrm{Disagreements}\right)\right)\kern0.5em \times \kern0.5em 100 $$

• f. RPMA—Responses per Minute Agreement

  • 1. Divide sessions into successive 60-s intervals.

  • 2. An agreement is scored for an interval if both observers record the same number of responses (e.g., both recorded zero responses; both recorded three responses).

  • 3. The total agreements and disagreements for a session is then put into the following formula:

    $$ \left(\mathrm{Number}\ \mathrm{of}\ \mathrm{agreements}\kern0.5em /\kern0.5em \left(\mathrm{Number}\ \mathrm{of}\ \mathrm{agreements}\kern0.5em +\kern0.5em \mathrm{Number}\ \mathrm{of}\ \mathrm{disagreements}\right)\right)\kern0.5em \times \kern0.5em 100. $$

Duration Behavior Reliability Calculations

• a. EIA—Exact Interval Agreement

  • 1. Divide sessions into successive 10-s intervals.

  • 2. An agreement is scored if both observers recorded exactly the same duration (rounded to the nearest 1 s) of the target response in that interval (e.g., both recorded 0 s; both recorded 3 s).

  • 3. The formula for exact agreement is as follows:

    $$ \left(\mathrm{Agreements}\kern0.5em /\kern0.5em \left(\mathrm{Agreements} + \mathrm{Disagreements}\right)\right)\kern0.5em \times \kern0.5em 100. $$

• b. PIA—Partial Interval Agreement

  • 1. Divide sessions into successive 10-s intervals.

  • 2. For each interval, the lower number of responses recorded is divided by the higher number. If both observers record zero responses, a ratio of 1.0 is substituted.

  • 3. The ratios are summed and divided by the number intervals in the session and then multiplied by 100.

    $$ \left(\mathrm{Sum}\ \mathrm{of}\ \mathrm{ratios}\kern0.5em /\kern0.5em \mathrm{Number}\ \mathrm{of}\ \mathrm{intervals}\right)\kern0.5em \times \kern0.5em 100 $$

• c. PMA—Partial Minute Agreement

  • 1. Divide sessions into successive 10-s intervals.

  • 2. For each interval, the lower number of responses recorded is divided by the higher number. If both observers record zero responses, a ratio of 1.0 is substituted.

  • 3. The ratios are summed and divided by the number intervals in the session and then multiplied by 100.

    $$ \left(\mathrm{Sum}\ \mathrm{of}\ \mathrm{ratios}\kern0.5em /\kern0.5em \mathrm{Number}\ \mathrm{of}\ \mathrm{intervals}\right)\kern0.5em \times \kern0.5em 100 $$

Appendix B

Program Validation

BDataPro was tested in order to ensure that (1) the timing is accurate and stable, (2) frequency and duration event recording is accurate, (3) the summary statistics are accurate, (4) the reliability coefficients are accurate, and (5) the program functions consistently across different makes and models of Windows© PC computers.

Testing should occur under conditions similar to those under which the program will be used. For example, users of a data collection program may also have other programs running simultaneously (e.g., web browser and word processor). If other programs may be running at the same time as data collection, then these programs should also be running during testing. What follows is a description of several types of tests that ensure the program is working correctly.

Testing for Accurate Timing, Event Recording, and Data analysis

Accurate recording of behavior and its occurrence in time are fundamental for the collection and analysis of behavioral data. It is critical to confirm that program keeps track of time correctly and consistently. Timing should remain accurate over long periods, independent of how many events are recorded. This can be confirmed with two types of tests referred to here as the “extended timer” and the “events timer” tests. Tests of timing accuracy require a previously validated, independent, timing device (e.g., stopwatch) with which to compare program timing.

Extended Timer Test

The extended timer test involves simply starting the data collection program timer and a stopwatch simultaneously. This test should be run for a period of time that exceeds the duration of observational periods for which the data collection program will likely be used. Both timers should indicate the same amount of time having passed at the end of the period.

Events Timer Test

The events timer test is designed to ensure that entering events (i.e., pressing a frequency or duration key) does not alter timing accuracy. For this test, two computers equipped with the data collection program can be used and a session should be run that exceeds the likely duration of observations periods for which the program will be used. The data collection program is started on both computers at the same time. For one of the computers, numerous events are recorded while the other computer simply runs the program. Both timers should remain synchronized regardless of the number of events recorded.

Frequency and Duration Events Test

This test is designed to examine whether the program output accurately reflects what frequency and duration events were actually entered (i.e., keystrokes). A known number of events should be recorded, across multiple frequency and duration keys, into a computer running BDataPro. The session summary data should correspond to the recorded events.

Summary Statistics Output Test

The purpose of this test is to examine the accuracy of calculations of summary statistics across multiple duration and frequency keys. Specifically, this test ensures accuracy of records concerning the total time each duration key is activated, the number of duration key bouts, the percentage of intervals during which a duration key is active, the percentage of intervals each frequency key is pressed, and responses per minute for each frequency key.

Reliability Tests

Calculation of inter-observer agreement by independent data collectors is a critical function of any observational data collection program. Examining the accuracy of calculated reliability coefficients should show that both agreement and disagreement of observations between two observers are correctly computed.

A straightforward method of ensuring that reliability between two observers whose data are in agreement is used to compare data from a two identical sessions in which multiple frequency and duration events were recorded. This test involves simply copying and relabeling a data file as a reliability file (simulating a data file generated from a second observer, but with known 100 % reliability). When these two files are compared, the program should indicate that all reliability coefficients are at 100 % agreement. Testing for accurate reliability calculations when observers disagree can be accomplished by comparing data files scored differently. Reliability coefficients for these two files are computed manually and compared to the program calculations. The manual calculations should correspond to those of the program.