Dirfile support

Support overview

As of mas/trunk/:r384, MAS supports writing of dirfiles. For now we are using a very basic dirfile standard, compatible with older versions of KST as well more recent versions of libgetdata.

Consult the sourceforge page for dirfile libraries and documentation:

http://getdata.sourceforge.net/

Although MAS can write dirfiles, we do not currently provide a facility for reading dirfiles into python or IDL, as we do for MCE flatfiles. We recommend that flatfiles be used for tuning and diagnostic data, with dirfiles only used for long acquisitions that will be analyzed off-line or by non-MAS software.

dirfiles are very useful for large, multi-rate data sets. They are not ideal in all circumstances, however.

Invocation

To acquire data into a dirfile, pass the --dirfile option to mce_run:

mce_run 10000 dirfile_0526 s --dirfile

This will create a dirfile at $MAS_DATA/dirfile_0526/ , and acquire 10000 frames from all readout card channels.

Please note the following:

• currently the "--dirfile" option must be listed after the three mandatory arguments.
• the number of "samples" requested should be a multiple of 100, as this is the samples/frame ratio specified in the MCE dirfile format file.

System configuration

To prevent crazy programs from bringing down the system, Linux controls the maximum number of files that any one shell can have open. The dirfile writer needs to keep a file open for each data channel, and for MCE data this could be ~2000 separate files. If the OS limit is too low to accommodate this, we need to increase it.

To check current OS limit, run ulimit:

mhasse@gamow:~$ ulimit -n
1024

This is too small. To increase it, edit /etc/security/limits.conf . Add / edit lines for item "nofile":

*               soft    nofile          2048

After reboot, you should get

mhasse@gamow:~$ ulimit -n
2048

Dirfile fields

Raw fields

The dirfile consists of raw fields and derived fields. For MCE data, the raw fields consist of fields from the frame header:

status
frame_ctr
row_len
num_rows_reported
data_rate
address0_ctr
header_version
ramp_value
ramp_addr
num_rows
sync_box_num
runfile_id
userfield

as well as the 32 bit words returned for each detector channel:

tesdatar00c00
tesdatar00c01
...

Note that MAS will determine exactly what channels are returning data and name the fields appropriately. So when reading RC2 only with readout_row_index=4 and num_rows_reported=1, the output fields will be

tesdatar04c08
tesdatar04c09
tesdatar04c10
tesdatar04c11
tesdatar04c12
tesdatar04c13
tesdatar04c14
tesdatar04c15

Derived fields

Based on the Data Mode of each readout card, derived signals are extracted from the raw data and made available to libgetdata users. The field extraction and conversion is accomplished according to the same rules used by mce_data.py and mas_data.pro , i.e. the data are scaled to some reference level for that signal. See the discussion in the Data Mode page.

The field names for error, feedback, filtered feedback, and phi0 counter data are, respectively (for r00c00)

error_r00c00
fb_r00c00
filt_r00c00
fj_r00c00

For some conversions, it is necessary to first extract the bitfield from the raw MCE word and then scale it. In these cases an intermediate field will be created, e.g.:

INTER_filt_r00c00

Such fields can be ignored.

Warning about signed fields in mixed-mode data

Currently the dirfile standard does not support sign extension of bit-field data. The result is that signed signals from mixed-mode MCE data will appear as positive numbers, with 'negative' values -D represented by the large positive number 2^n - D. For example, in data mode 4 the error signal is encoded in the lowest-order 14 bits of the MCE word. An error reading of 12 gets encoded as 00000000001100 in binary and would be extracted as 12, but an error reading of -12 would be encoded with the 14 binary digits 11111111110100, and upon extraction would end up as 2^14 - 12 = 16372.

Sign-related issues can be corrected in down-stream software; there is no information loss. But something like KST will not naturally 'do the right thing' here, so the time-stream and spectrum plots will be awful.