Testing

eCos USB Slave Support
Prev

Introduction

The support for USB testing provided by the eCos USB common slave package is somewhat different in nature from the kind of testing used in many other packages. One obvious problem is that USB tests cannot be run on just a bare target platform: instead the target platform must be connected to a suitable USB host machine, and that host machine must be running appropriate software for the test code to interact with. This is very different from say a kernel test which typically will have no external dependencies. Another important difference between USB testing and say a C library strcmp test is sensitivity to timing and to hardware boundary conditions: although a simple test case that just performs a small number of USB transfers is better than no testing at all, it should also be possible to run tests for hours or days on end, under a variety of loads. In order to provide the required functionality the basic architecture of the USB testing support is as follows:

There is a single target-side program usbtarget. By default when this is run on a target platform it will appear to do nothing. In fact it is waiting to be contacted by another program usbhost which will tell it what test or tests to run. usbtarget provides mechanisms for running a wide range of tests.
usbtarget is a generic program, but USB testing depends to some extent on the functionality provided by the hardware. For example there is no point in testing bulk transmits to endpoint 12 if the target hardware does not support an endpoint 12. Therefore each USB device driver should supply information about what the hardware is actually capable of, in the form of an array of usbs_testing_endpoint data structures.
There is a single host-side program usbhost, which acts as a counterpart to usbtarget. Again usbhost has no built-in knowledge of the test or tests that are supposed to run, it only provides mechanisms for running a wide range of tests. On start-up usbhost will search the USB bus for hardware running the target-side program, specifically a USB device that identifies itself as the product "Red Hat eCos USB test".
usbhost contains a Tcl interpreter, and will execute any Tcl scripts specified on the command line together with appropriate arguments. The Tcl interpreter has been extended with various commands such as usbtest::bulktest, so the script can perform the desired test or tests.
Adding a new test simply involves writing a short Tcl script that invokes the appropriate USB-specific commands. Running multiple tests involves passing appropriate arguments to usbhost, or alternatively writing a single script that just invokes other scripts.

The current implementation of usbhost depends heavily on functionality provided by the Linux kernel and in particular the usbdevfs support. It uses /proc/bus/usb/devices to find out what devices are attached to the bus, and will then access the device by opening /proc/bus/usb/xxx/yyy and performing ioctl operations. This allows USB testing to take place without having to write a new host-side device driver, but getting the code working on host machines not running Linux would obviously be problematical.

Building and Running the Target-side Code

The target-side component of the USB testing software consists of a single program usbtarget which contains support for a range of different tests, under the control of host-side software. This program is not built by default alongside other eCos test cases since it will only operate in certain environments, specifically when the target board's connector is plugged into a Linux host, and when the appropriate host-side software has been installed on that host. Instead the user must enable a configuration option CYGBLD_IO_USB_SLAVE_USBTEST to add the program to the list of tests for the current configuration.

Starting the usbtarget program does not require anything unusual, so it can be run in a normal gdb session just like any eCos application. After initialization the program will wait for activity from the host. Depending on the hardware, the Linux host will detect that a new USB peripheral is present on the bus either when the usbtarget initialization is complete or when the cable between target and host is connected. The host will perform the normal USB enumeration sequence and discover that the peripheral does not match any known vendor or product id and that there is no device driver for "Red Hat eCos USB test", so it will ignore the peripheral. When the usbhost program is run on the host it will connect to the target-side software, and testing can now commence.

Building and Running the Host-side Code

Note: In theory the host-side software should be built when the package is installed in the component repository, and removed when a package is uninstalled. The current eCos administration tool does not provide this functionality.

The host-side software should be built via the usual sequence of "configure/make/make install". It can only be built on a Linux host and the configure script contains an explicit test for this. Because the eCos component repository should generally be treated as a read-only resource the configure script will also prevent you from trying to build inside the source tree. Instead a separate build tree is required. Hence a typical sequence for building the host-side software would be as follows:

$ mkdir usbhost_build
$ cd usbhost_build
$ <repo>packages/io/usb/slave/current/host/configure   <args> 
$ make
<output from make>
$ su 
$ make install
<output from make install>
$

: The location of the eCos component repository should be substituted for <repo>.
: If the package has been obtained via CVS or anonymous CVS then the package version will be current, as per the example. If instead the package has been obtained as part of a full eCos release or as a separate .epk file then the appropriate package version should be used instead of current.
: The configure script takes the usual arguments such as --prefix= to specify where the executables and support files should be installed. The only other parameter that some users may wish to specify is the location of a suitable Tcl installation. By default usbhost will use the existing Tcl installation in /usr, as provided by your Linux distribution. An alternative Tcl installation can be specified using the parameter --with-tcl=, or alternatively using some combination of --with-tcl-include, --with-tcl-lib and --with-tcl-version.
: One of the host-side executables that gets built, usbchmod, needs to be installed with suid root privileges. Although the Linux kernel makes it possible for applications to perform low-level USB operations such as transmitting bulk packets, by default access to this functionality is restricted to programs with superuser privileges. It is undesirable to run a complex program such as usbhost with such privileges, especially since the program contains a general-purpose Tcl interpreter. Therefore when usbhost starts up and discovers that it does not have sufficient access to the appropriate entries in /proc/bus/usb, it spawns an instance of usbchmod to modify the permissions on these entries. usbchmod will only do this for a USB device "Red Hat eCos USB test", so installing this program suid root should not introduce any security problems.

During make install the following actions will take place:

usbhost will be installed in /usr/local/bin, or some other bin directory if the default location is changed at configure-time using a --prefix= or similar option. It will be installed as the executable usbhost_<version>, for example usbhost_current, thus allowing several releases of the USB slave package to co-exist. For convenience a symbolic link from usbhost to this executable will be created, so users can just run usbhost to access the most recently-installed version.
usbchmod will be installed in /usr/local/libexec/ecos/io_usb_slave_<version>. This program should only be run by usbhost, not invoked directly, so it is not placed in the bin directory. Again the presence of the package version in the directory name allows multiple releases of the package to co-exist.
A Tcl script usbhost.tcl will get installed in the same directory as usbchmod. This Tcl script is loaded automatically by the usbhost executable.
A number of additional Tcl scripts, for example list.tcl will get installed alongside usbhost.tcl. These correspond to various test cases provided as standard. If a given test case is specified on the command line and cannot be found relative to the current directory then usbhost will search the install directory for these test cases.
Note: Strictly speaking installing the usbhost.tcl and other Tcl scripts below the libexec directory deviates from standard practice: they are architecture-independent data files so should be installed below the share subdirectory. In practice the files are sufficiently small that there is no point in sharing them, and keeping them below libexec simplifies the host-side software somewhat.

The usbhost should be run only when there is a suitable target attached to the USB bus and running the usbtarget program. It will search /proc/bus/usb/devices for an entry corresponding to this program, invoke usbchmod if necessary to change the access rights, and then interact with usbtarget over the USB bus. usbhost should be invoked as follows:

$ usbhost [-v|--version] [-h|--help] [-V|--verbose] <test> [<test parameters>]

The -v or --version option will display version information for usbhost including the version of the USB slave package that was used to build the executable.
The -h or --help option will display usage information.
The -V or --verbose option can be used to obtain more information at run-time, for example some output for every USB transfer. This option can be repeated multiple times to increase the amount of output.
The first argument that does not begin with a hyphen specifies a test that should be run, in the form of a Tcl script. For example an argument of list.tcl will cause usbhost to look for a script with that name, adding a .tcl suffix if necessarary, and run that script. usbhost will look in the current directory first, then in the install tree for standard test scripts provided by the USB slave package.
Some test scripts may want their own parameters, for example a duration in seconds. These can be passed on the command line after the name of the test, for example usbhost mytest 60.

Writing a Test

Each test is defined by a Tcl script, running inside an interpreter provided by usbhost. In addition to the normal Tcl functionality this interpreter provides a number of variables and functions related to USB testing. For example there is a variable bulk_in_endpoints that lists all the endpoints on the target that can perform bulk IN operations, and a related array bulk_in which contains information such as the minimum and maximum packets sizes. There is a function bulktest which can be used to perform bulk tests on a particular endpoint. A simple test script aimed at specific hardware could ignore the information variables since it would know exactly what USB hardware is available on the target, whereas a general-purpose script would use the information to adapt to the hardware capabilities.

To avoid namespace pollution all USB-related Tcl variables and functions live in the usbtest:: namespace. Therefore accessing requires either explicitly including the namespace any references, for example $usbtest::bulk_in_endpoints, or by using Tcl's namespace import facility.

A very simple test script might look like this:

usbtest::bulktest 1 out 4000
usbtest::bulktest 2 in  4000
if { [usbtest::start 60] } {
    puts "Test successful"
} else
    puts "Test failed"
    foreach result $usbtest::results {
        puts $result
    }
}

This would perform a test run involving 4000 bulk transfers from the host to the target's endpoint 1, and concurrently 4000 bulk transfers from endpoint 2. Default settings for packet sizes, contents, and delays would be used. The actual test would not start running until usbtest is invoked, and it is expected that the test would complete within 60 seconds. If any failures occur then they are reported.

Available Hardware

Each target-side USB device driver provides information about the actual capabilities of the hardware, for example which endpoints are available. Strictly speaking it provides information about what is actually supported by the device driver, which may be a subset of what the hardware is capable of. For example, the hardware may support isochronous transfers on a particular endpoint but if there is no software support for this in the driver then this endpoint will not be listed. When usbhost first contacts the usbtarget program running on the target platform, it obtains this information and makes it available to test scripts via Tcl variables:

bulk_in_endpoints

This is a simple list of the endpoints which can support bulk IN transfers. For example if the target-side hardware supports these transfers on endpoints 3 and 5 then the value would be "3 5" Typical test scripts would iterate over the list using something like:

  if { 0 != [llength $usbtest::bulk_in_endpoints] } {
      puts"Bulk IN endpoints: $usbtest::bulk_in_endpoints"
      foreach endpoint $usbtest:bulk_in_endpoints {
          …
      }
  }

bulk_in()

This array holds additional information about each bulk IN endpoint. The array is indexed by two fields, the endpoint number and one of min_size, max_size, max_in_padding and devtab:

min_size: This field specifies a lower bound on the size of bulk transfers, and will typically will have a value of 1.
Note: The typical minimum transfer size of a single byte is not strictly speaking correct, since under some circumstances it can make sense to have a transfer size of zero bytes. However current target-side device drivers interpret a request to transfer zero bytes as a way for higher-level code to determine whether or not an endpoint is stalled, so it is not actually possible to perform zero-byte transfers. This issue will be addressed at some future point.
max_size: This field specifies an upper bound on the size of bulk transfers. Some target-side drivers may be limited to transfers of say 0x0FFFF bytes because of hardware limitations. In practice the transfer size is likely to be limited primarily to limit memory consumption of the test code on the target hardware, and to ensure that tests complete reasonably quickly. At the time of writing transfers are limited to 4K.
max_in_padding: On some hardware it may be necessary for the target-side device driver to send more data than is actually intended. For example the SA11x0 USB hardware cannot perform bulk transfers that are an exact multiple of 64 bytes, instead it must pad such transfers with an extra byte and the host must be ready to accept and discard this byte. The max_in_padding field indicates the amount of padding that is required. The low-level code inside usbhost will use this field automatically, and there is no need for test scripts to adjust packet sizes for padding. The field is provided for informational purposes only.
devtab: This is a string indicating whether or not the target-side USB device driver supports access to this endpoint via entries in the device table, in other words through conventional calls like open and write. Some device drivers may only support low-level USB access because typically that is what gets used by USB class-specific packages such as USB-ethernet. An empty string indicates that no devtab entry is available, otherwise it will be something like "/dev/usbs2w".

Typical test scripts would access this data using something like:

  foreach endpoint $usbtest:bulk_in_endpoints {
      puts "Endpoint $endpoint: "
      puts "    minimum transfer size $usbtest::bulk_in($endpoint,min_size)"
      puts "    maximum transfer size $usbtest::bulk_in($endpoint,max_size)"
      if { 0 == $usbtest::bulk_in($endpoint,max_in_padding) } {
          puts "    no IN padding required"
      } else {
          puts "    $usbtest::bulk_in($endpoint,max_in_padding) bytes of IN padding required"
      }
      if { "" == $usbtest::bulk_in($endpoint,devtab) } {
          puts "    no devtab entry provided"
      } else {
          puts "    corresponding devtab entry is $usbtest::bulk_in($endpoint,devtab)"
      }
  }

bulk_out_endpoint

This is a simple list of the endpoints which can support bulk OUT transfers. It is analogous to bulk_in_endpoints.

bulk_out()

This array holds additional information about each bulk OUT endpoint. It can be accessed in the same way as bulk_in(), except that there is no max_in_padding field because that field only makes sense for IN transfers.

control()

This array holds information about the control endpoint. It contains two fields, min_size and max_size. Note that there is no variable control_endpoints because a USB target always supports a single control endpoint 0. Similarly the control array does not use an endpoint number as the first index because that would be redundant.

isochronous_in_endpoints and isochronous_in()

These variables provide the same information as bulk_in_endpoints and bulk_in, but for endpoints that support isochronous IN transfers.

isochronous_out_endpoints and isochronous_out()

These variables provide the same information as bulk_out_endpoints and bulk_out, but for endpoints that support isochronous OUT transfers.

interrupt_in_endpoints and interrupt_in()

These variables provide the same information as bulk_in_endpoints and bulk_in, but for endpoints that support interrupt IN transfers.

interrupt_out_endpoints and interrupt_out()

These variables provide the same information as bulk_out_endpoints and bulk_out, but for endpoints that support interrupt OUT transfers.

Testing Bulk Transfers

The main function for initiating a bulk test is usbtest::bulktest. This takes three compulsory arguments, and can be given a number of additional arguments to control the exact behaviour. The compulsory arguments are:

endpoint: This specifies the endpoint to use. It should correspond to one of the entries in usbtest::bulk_in_endpoints or usbtest::bulk_out_endpoints, depending on the transfer direction.
direction: This should be either in or out.
number of transfers: This specifies the number of transfers that should take place. The testing software does not currently support the concept of performing transfers for a given period of time because synchronising this on both the host and a wide range of targets is difficult. However it is relatively easy to work out the approximate time a number of bulk transfers should take place, based on a typical bandwidth of 1MB/second and assuming say a 1ms overhead per transfer. Alternatively a test script could perform a small initial run to determine what performance can actually be expected from a given target, and then use this information to run a much longer test.

Additional arguments can be used to control the exact transfer. For example a txdelay+ argument can be used to slowly increase the delay between transfers. All such arguments involve a value which can be passed either as part of the argument itself, for example txdelay+=5, or as a subsequent argument, txdelay+ 5. The possible arguments fall into a number of categories: data, I/O mechanism, transmit size, receive size, transmit delay, and receive delay.

Data

An obvious parameter to control is the actual data that gets sent. This can be controlled by the argument format which can take one of five values: none, bytefill, intfill, byteseq and wordseq. The default value is none.

none: The transmit code will not attempt to fill the buffer in any way, and the receive code will not check it. The actual data that gets transferred will be whatever happened to be in the buffer before the transfer started.
bytefill: The entire buffer will be filled with a single byte, as per memset.
intfill: The buffer will be treated as an array of 32-bit integers, and will be filled with the same integer repeated the appropriate number of times. If the buffer size is not a multiple of four bytes then the last few bytes will be set to 0.
byteseq: The buffer will be filled with a sequence of bytes, generated by a linear congruential generator. If the first byte in the buffer is filled with the value x, the next byte will be (m*x)+i. For example a sequence of slowly incrementing bytes can be achieved by setting both the multiplier and the increment to 1. Alternatively a pseudo-random number sequence can be achieved using values 1103515245 and 12345, as per the standard C library rand function. For convenience these two constants are available as Tcl variables usbtest::MULTIPLIER and usbtest::INCREMENT.
wordseq: This acts like byteseq, except that the buffer is treated as an array of 32-bit integers rather than as an array of bytes. If the buffer is not a multiple of four bytes then the last few bytes will be filled with zeroes.

The above requires three additional parameters data1, data* and data+. data1 specifies the value to be used for byte or word fills, or the first number when calculating a sequence. The default value is 0. data* and data+ specify the multiplier and increment for a sequence, and have default values of 1 and 0 respectively. For example, to perform a bulk transfer of a pseudo-random sequence of integers starting with 42 the following code could be used:

bulktest 2 IN 1000 format=wordseq data1=42 \
    data* $usbtest::MULTIPLIER data+ $usbtest::INCREMENT

The above parameters define what data gets transferred for the first transfer, but a test can involve multiple transfers. The data format will be the same for all transfers, but it is possible to adjust the current value, the multiplier, and the increment between each transfer. This is achieved with parameters data1*, data1+, data**, data*+, data+*, and data++, with default values of 1 for each multiplier and 0 for each increment. For example, if the multiplier for the first transfer is set to 2 using data*, and arguments data** 2 and data*+ -1 are also supplied, then the multiplier for subsequent transfers will be 3, 5, 9, ….

Note: Currently it is not possible for a test script to send specific data, for example a specific sequence of bytes captured by a protocol analyser that caused a problem. If the transfer was from host to target then the target would have to know the exact sequence of bytes to expect, which means transferring data over the USB bus when that data is known to have caused problems in the past. Similarly for target to host transfers the target would have to know what bytes to send. A possible future extension of the USB testing support would allow for bounce operations, where a given message is first sent to the target and then sent back to the host, with only the host checking that the data was returned correctly.

I/O Mechanism

On the target side USB transfers can happen using either low-level USB calls such as usbs_start_rx_buffer, or by higher-level calls which go through the device table. By default the target-side code will use the low-level calls. If it is desired to test the higher-level calls instead, for example because those are what the application uses, then that can be achieved with an argument mechanism=devtab.

Transmit Size

The next set of arguments can be used to control the size of the transmitted buffer: txsize1, txsize>=, txsize<= txsize*, txsize/, and txsize+.

txsize1 determines the size of the first transfer, and has a default value of 32 bytes. The size of the next transfer is calculated by first multiplying by the txsize* value, then dividing by the txsize/ value, and finally adding the txsize+ value. The defaults for these are 1, 1, and 0 respectively, which means that the transfer size will remain unchanged. If for example the transfer size should increase by approximately 50 per cent each time then suitable values might be txsize* 3, txsize/ 2, and txsize+ 1.

The txsize>= and txsize<= arguments can be used to impose lower and upper bounds on the transfer. By default the min_size and max_size values appropriate for the endpoint will be used. If at any time the current size falls outside the bounds then it will be normalized.

Receive Size

The receive size, in other words the number of bytes that either host or target will expect to receive as opposed to the number of bytes that actually get sent, can be adjusted using a similar set of arguments: rxsize1, rxsize>=, rxsize<=, rxsize*, rxsize/ and rxsize+. The current receive size will be adjusted between transfers just like the transmit size. However when communicating over USB it is not a good idea to attempt to receive less data than will actually be sent: typically neither the hardware nor the software will be able to do anything useful with the excess, so there will be problems. Therefore if at any time the calculated receive size is less than the transmit size, the actual receive will be for the exact number of bytes that will get transmitted. However this will not affect the calculations for the next receive size.

The default values for rxsize1, rxsize*, rxsize/ and rxsize+ are 0, 1, 1 and 0 respectively. This means that the calculated receive size will always be less than the transmit size, so the receive operation will be for the exact number of bytes transmitted. For some USB protocols this would not accurately reflect the traffic that will happen. For example with USB-ethernet transfer sizes will vary between 16 and 1516 bytes, so the receiver will always expect up to 1516 bytes. This can be achieved using rxsize1 1516, leaving the other parameters at their default values.

For target hardware which involves non-zero max_in_padding, on the host side the padding will be added automatically to the receive size if necessary.

Transmit and Receive Delays

Typically during the testing there will be some minor delays between transfers on both host and target. Some of these delays will be caused by timeslicing, for example another process running on the host, or a concurrent test thread running inside the target. Other delays will be caused by the USB bus itself, for example activity from another device on the bus. However it is desirable that test cases be allowed to inject additional and somewhat more controlled delays into the system, for example to make sure that the target behaves correctly even if the target is not yet ready to receive data from the host.

The transmit delay is controlled by six parameters: txdelay1, txdelay*, txdelay/, txdelay+, txdelay>= and txdelay<=. The default values for these are 0, 1, 1, 0, 0 and 1000000000 respectively, so that by default transmits will happen as quickly as possible. Delays are measured in nanoseconds, so a value of 1000000 would correspond to a delay of 0.001 seconds or one millisecond. By default delays have an upper bound of one second. Between transfers the transmit delay is updated in much the same was as the transfer sizes.

The receive delay is controlled by a similar set of six parameters: rxdelay1, rxdelay*, rxdelay/, rxdelay+, rxdelay>= and rxdelay<=. The default values for these are the same as for transmit delays.

The transmit delay is used on the side which sends data over the USB bus, so for a bulk IN transfer it is the target that sends data and hence sleeps for the specified transmit delay, while the host receives data sleeps for the receive delay. For an OUT transfer the positions are reversed.

It should be noted that although the delays are measured in nanoseconds, the actual delays will be much less precise and are likely to be of the order of milliseconds. The exact details will depend on the kernel clock speed.

Other Types of Transfer

Support for testing other types of USB traffic such as isochronous transfers is not yet implemented.

Starting a Test and Collecting Results

A USB test script should prepare one or more transfers using appropriate functions such as usbtest::bulktest. Once all the individual tests have been prepared they can be started by a call to usbtest::start. This takes a single argument, a maximum duration measured in seconds. If all transfers have not been completed in the specified time then any remaining transfers will be aborted.

usbtest::start will return 1 if all the tests have succeeded, or 0 if any of them have failed. More detailed reports will be stored in the Tcl variable usbtests::results, which will be a list of string messages.

Existing Test Scripts

A number of test scripts are provided as standard. These are located in the host subdirectory of the common USB slave package, and will be installed as part of the process of building the host-side software. When a script is specified on the command line usbhost will first search for it in the current directory, then in the install tree. Standard test scripts include the following:

list.tcl: This script simply displays information about the capabilities of the target platform, as provided by the target-side USB device driver. It can help with tracking down problems, but its primary purpose is to let users check that everything is working correctly: if running usbhost list.tcl outputs sensible information then the user knows that the target side is running correctly and that communication between host and target is possible.
verbose.tcl: The target-side code can provide information about what is happening while tests are prepared and run. This facility should not normally be used since the extra I/O involved will significantly affect the behaviour of the system, but in some circumstances it may prove useful. Since an eCos application cannot easily be given command-line arguments the target-side verbosity level cannot be controlled using -V or --verbose options. Instead it can be controlled from inside gdb by changing the integer variable verbose. Alternatively it can be manipulated by running the test script verbose.tcl. This script takes a single argument, the desired verbosity level, which should be a small integer. For example, to disable target-side run-time logging the command usbhost verbose 0 can be used.

Possible Problems

If all transfers succeed within the specified time then both host and target remain in synch and further tests can be run without problem. However, if at any time a failure occurs then things get more complicated. For example, if the current test involves a series of bulk OUT transfers and the target detects that for one of these transfers it received less data than was expected then the test has failed, and the target will stop accepting data on this endpoint. However the host-side software may not have detected anything wrong and is now blocked trying to send the next lot of data.

The test code goes to considerable effort to recover from problems such as these. On the host-side separate threads are used for concurrent transfers, and on the target-side appropriate asynchronous I/O mechanisms are used. In addition there is a control thread on the host that checks the state of all the main host-side threads, and the state of the target using private control messages. If it discovers that one side has stopped sending or receiving data because of an error and the other side is blocked as a result, it will set certain flags and then cause one additional transfer to take place. That additional transfer will have the effect of unblocking the other side, which then discovers that an error has occurred by checking the appropriate flags. In this way both host and target should end up back in synch, and it is possible to move on to the next set of tests.

However, the above assumes that the testing has not triggered any serious hardware conditions. If instead the target-side hardware has been left in some strange state so that, for example, it will no longer raise an interrupt for traffic on a particular endpoint then recovery is not currently possible, and the testing software will just hang.

A possible future enhancement to the testing software would allow the host-side to raise a USB reset signal whenever a failure occurs, in the hope that this would clear any remaining problems within the target-side USB hardware.

Prev	Home
Writing a USB Device Driver