Rectangle Mode Data
Rectangle Mode refers to the sampling of a subset of the full array, typically at a kHz data rate. The data from the Readout Cards is packed into output frames efficiently to minimize overhead.
Rectangle mode eliminates strain on the acquisition system during fast acquisitions by eliminating read-out overhead. This means that the 'difficulty' of read-out is a fairly linear function of (sampling frequency) x (number of detectors read out) rather than depending on each.
The hardware limit on readout is roughly 4 MB/s. Since MCE data are 32-bit words, this permits 106 detector samples to be read out per second.
So the maximum number of detectors D that can be sampled at some frequency F is
D = (1 MHz) / F
You can't sample at 1 MHz. But you could probably do a detector or two at 250 kHz.
Contents
Configuration
The steps for configuring a rectangle mode acquisition are:
- Set the multiplexing rate using num_rows and row_len
- If necessary, adjust row_order to mux the channels of interest (you can only read out rows you are multiplexing...). If you do not intend to re-tune the array with the adjusted num_rows / row_order, you must also change the first entries of rc# adc_offset# to correspond to the desired rows and, for fast SQ2 switching, change row_order on BC2 (or BAC).
- Identify a rectangle of detectors on the readout card to read out by setting the RC parameters
- Set the size of the readout frame by setting the CC parameters
- Set the CC decimation factor data_rate so that the RC rectangles will stack perfectly into the CC readout frame.
- e.g. suppose
- RC: num_rows_reported x num_cols_reported = 1x4
- CC: num_rows_reported x num_cols_reported = 33x8
- then we set
- CC: data_rate = (33 x 8) / (1 x 4) = 66
- e.g. suppose
Readout limitations
Rectangle mode configuration should respect the following hardware limits:
- Total data rate must not exceed 4 MB/s
- Read-out frequency must not exceed 20 kHz
- The multiplexing frame rate must not be set so fast that the MCE cannot communicate with the readout cards. This limit is probably in the 100s of kHz.
- The buffer on each RC can hold 2048 words of data. The readout frame size, per readout card, must be kept somewhat below this limit.
Other things to remember:
- num_rows ≥ 2 is encouraged.
- you can't sample a detector faster than it is being visited in the multiplexing cycle.
- In RC firmware prior to 5.1.7, the following condition has to be met:
(230 + 2 * cc num_rows_reported * cc num_cols_reported) < (num_rows * row_len)
otherwise, the data is corrupted. This problem is fixed in RC firmware 5.1.7 (See Readout Card firmware)
Checking MCE configuration using rect_check.py
Use the utility rect_check.py to verify that data will be / were acquired reasonably.
e.g. To check the state of the MCE (requires python MCE bindings), run
rect_check.py --mce
To analyze the data framing in a runfile:
rect_check.py my_data.run
(Depending on stuff, you might need to invoke rect_check as:
$MAS_PYTHON/rect_check.py
or even
python $MAS_PYTHON/rect_check.py
Sorry.)
The output looks like this:
MCE configuration: cc_rcs ( cc, rcs_to_report_data ): 60 cc_dec ( cc, data_rate ): 256 cc_nmux ( cc, num_rows ): 33 cc_cmux ( cc, row_len ): 100 cc_nr ( cc, num_rows_reported ): 32 cc_nc ( cc, num_cols_reported ): 8 rc_nmux ( rca, num_rows ): 33 rc_cmux ( rca, row_len ): 100 rc_nr ( rca, num_rows_reported ): 1 rc_nc ( rca, num_cols_reported ): 1 rc_r0 ( rca, readout_row_index ): 0 rc_c0 ( rca, readout_col_index ): 0 Framing: RC storage per mux cycle: 1 CC read-out per frame: 256 CC decimation: 256 Sanity Summary: Contiguous (for high-rate readout)? yes Complete (all-detector readout)? no Bizarro (a bad thing)? no Bounded (a good thing)? yes Timing: Mux freq: 15151.00 Mean sampling freq: 15151.00 Read-out freq: 59.00 Data volume: Frame size (bytes/RC): 1200 Data rate (MB/s/RC): 0.07
The program has notions of what is "reasonable", and these are expressed in the lines "Contiguous?", "Complete?", "Bizarro?", and "Bounded?":
- 'Contiguous': all detectors that are read out will be read out at each multiplexing cycle
- When acquiring fast data from a subset of the detectors, at the multiplexing rate, you want "Contiguous = yes".
- 'Complete': all detectors that are multiplexed will be read out
- When acquiring normal data, you probably want "Complete = yes".
- 'Bizarro': the rectangle defined in the RC does not fit evenly into the readout frame.
- e.g. you have set up an 8x8 rectangle but you're reading CC frames that are 33x8. Don't do this. I'm not even sure what you'd get out.
- 'Bounded': the rectangle defined in the RC corresponds to detectors that are actually getting read/servoed. This basically just checks that rc# readout_row_index makes any sense at all given num_rows and num_rows_reported.
- This must be "yes" or your data could be weird.
Note that it is possible to be none of Complete, Contiguous, or Bizarro. For example, you could read out the first 33 of every 66 multiplexing cycles for 8 detectors, packing them efficiently into a 33 x 8 read-out frame. Maybe that should be called Bizarro. But right now, it isn't.
It is unlikely, but possible, to be both Contiguous and Complete. If you are Bizarro, you are neither Contiguous nor Complete. See Euler diagram below.
Data acquisition
Use "mce_run" in the usual way to acquire data. Note that the argument specifying the number of frames is always the number of 'read-out' frames, so for packed data the number of samples per detector will be higher.
e.g. for the above configuration
mce_run my_data 100 1
would acquire 100 read-out frames. Since the read-out frequency is 59 Hz, this would take about 2 seconds. Each readout frame contains 8 x 32 = 256 data from a single detector. So the resulting data stream contains 25600 samples.
Data file loading
Recent versions of mce_data.py can extract rectangle mode data from the packed frame structure. E.g. for the above case, extracting the data is as easy as
>>> d = MCEFile('my_data').Read()
>>> d.data.shape
(1, 25600)
Rather long explanation and justification of structure, from an e-mail
Yesterday Bryce and I were talking about paths towards a new rectangle readout mode. The following is a recommendation for the rough structure of the system that I think will be easy to manage and provide a maximum flexibility for backwards compatibility and future expansion.
The current rectangle mode is simpler than the planned one because the clock card and readout card need only agree on the number of data to readout, and no addditional waiting/synchronization is required between the clock card and readout card. The readout frame data sizes, internally and externally, are the same for everyone. The new mode is much more complex, because now the readout card has to accumulate several samples from each pixel into its internal buffer before sending the data to the clock card, and the clock card needs to ask for the right amount of data, which is probably some multiple of the rectangle size and is not necessarily a multiple of 8 or the number of multiplexing rows.
Anyway, the point is that here is a simple and flexible way to parametrize the new mode parameters and manage synchronization, and to provide flexibility and backwards compatibility.
1. The meaning of num_rows remains unchanged. It will always be a multiplexing parameter that all cards need to agree on.
2. The readout card has parameters (prepend "readout_" to all of these, if you want) "row_index", "num_rows", "col_index", "num_cols" that describe the rectangle of interest.
3. The clock card is programmed with a pair of numbers "readout_mult1" and "readout_mult2", that it uses to determine how many words of data to query the RCs for:
N_data = readout_mult1 * readout_mult2.
These numbers are analagous to "num_rows_reported" and "num_cols_reported", but the idea is to kill the idea of "rows" and "columns" in the readout frames because the payload size is not a simple product of n_rows * n_cols. The clock card continues to support "data rate", which indicates the period at which data queries are sent to the RCs. Software will handle the details of getting these factors right. Eventually we will only use one of mult?; the other is for backwards compat.
4. In the RC: at each mux frame, data from the desired rectangle is copied into a large buffer. The RC continues filling the buffer until the CC asks for data. When RC receives a request for N data from the CC, it returns the first N data in the buffer, and resets its write index to the start of the buffer. If the CC doesn't ask for data, and the RC buffer write index increments all the way past the end, the RC takes _no storage action_ in subsequent steps (i.e. it writes all new data to "/dev/null").
Setting up rectangle mode then proceeds as follows. Suppose the RC buffer size is 1600 and I want to readout a 4 x 33 patch of the array:
1. Software computes that 4x33 = 132 words fits into 1600 about 12 times. So it sets "cc data_rate" to 12, and the product of "mult1" and "mult2" to 12 * 132 = 1584. It also sets the readout card row and col rectangle parameters.
2. Software triggers frame acquisition. For the _first_ frame read out, the clock card and readout card may not be synchronized; i.e. the readout card's buffer almost certainly contains ancient, stale data. So the first frame is discarded. But since the first frame query sent the the buffer write index to 0, the data from subsequent frames is fresh and ordered. Because the clock card is asking for N data at the same rate that the RC can N-fill the buffer, this results in a contiguous data stream.
This scheme is simple because: 1. CC has only enough information to ask the readout cards for the right amount of data at the right times. 2. RC only needs to know the rectangle of interest, and it basically lets the CC manage the resetting of the buffer. 3. It is backwards-compatible with existing firmware; mult1 and mult2 can masquerade as num_rows_reported and "8" (num_cols_reported).
The scheme minimizes the amount of information that must be shared between the RC and CC, which provides flexiblity. With "mult" approach, the CC can remain truly ignorant as to what the RC is doing. In this scheme I have emphasized the rectangle mode features, but I also envision it being useful in raw mode firmware, since a lot of the features (such as a large, flexible RAM block in the RC) would seem useful in that context. (For raw mode you would need to add a lock to the buffer so that it could be read out with multiple queries.)
Script Differences Between v4.x.x and v5.x.x
This section describes the minimal configuration changes necessary to operate v5 firmware as though it were v4 firmware. If you use MAS, then stop reading now.
The differences between basic frame setup in v4.x.x and v5.x.x are described here. The data frame structure has been made more flexible. A user can read out an arbitrarily rectangular patch of the array. The patch-size is specified using new commands.
- In v4.x.x: The array patch-size is flexible in terms of the number of rows, and the row start index.
wb cc num_rows_reported 41 //power-on default = 41 wb rca readout_row_index 0 //power-on default = 0
- In v5.x.x: The array patch-size is now also flexible in terms of the number of columns, and the column start index. Note that matching parameters on the Readout Cards and the Clock Card must be consistent.
wb cc num_rows_reported 41 //power-on default as above wb cc num_cols_reported 8 //power-on default = 8 wb rca num_rows_reported 41 //power-on default = 41 wb rca num_cols_reported 8 //power-on default = 8 wb rca readout_row_index 0 //power-on default as above wb rca readout_col_index 0 //power-on default = 0
