Dirfile support

Support overview

As of mas/trunk/r384, MAS supports writing of dirfiles. The default dirfile version written by MAS has recently been upgraded to version Seven, to allow the use of signed bitfield extraction (see below). The writing of older versions is still supported, which can be used with older versions of KST, and which are still compatible with the modern GetData library.

Consult the sourceforge page for dirfile libraries and documentation:

http://getdata.sourceforge.net/

Although MAS can write dirfiles, we do not currently provide a facility for reading dirfiles into python or IDL, as we do for MCE flatfiles. The GetData library provides basic bindings for reading dirfiles in both IDL and Python, but these lack MAS specific benefits of reading flatfiles with MAS and MCE script. We recommend that flatfiles be used for tuning and diagnostic data, with dirfiles only used for long acquisitions that will be analyzed off-line or by non-MAS software.

Dirfiles are very useful for large, multi-rate data sets. They are not ideal in all circumstances, however.

Invocation

To acquire data into a dirfile, pass the --dirfile option to mce_run:

mce_run 10000 dirfile_0526 s --dirfile

This will create a dirfile at $MAS_DATA/dirfile_0526/ , and acquire 10000 frames from all readout card channels.

Please note the following:

• currently the "--dirfile" option must be listed after the three mandatory arguments.
• "file-sequenced" operation is supported as of MAS r624 + mce_script r912. Pass "--sequence <chunk_size>" to break the acquisition into pieces containing <chunk_size> frames. The pieces will be numbered as dirfile_0526.000, dirfile_0526.001, etc.

System configuration

To prevent crazy programs from bringing down the system, Linux controls the maximum number of files that any one shell can have open. The dirfile writer needs to keep a file open for each data channel, and for MCE data this could be ~2000 separate files. If the OS limit is too low to accommodate this, we need to increase it.

To check current OS limit, run ulimit:

mhasse@gamow:~$ ulimit -n
1024

This is too small. To increase it, edit /etc/security/limits.conf . Add / edit lines for item "nofile":

*               soft    nofile          2048
*               hard    nofile          4096

After reboot, you should get

mhasse@gamow:~$ ulimit -n
2048

Dirfile fields

Raw fields

The dirfile consists of raw fields and derived fields. For MCE data, the raw fields consist of fields from the frame header:

status
frame_ctr
row_len
num_rows_reported
data_rate
address0_ctr
header_version
ramp_value
ramp_addr
num_rows
sync_box_num
runfile_id
userfield

as well as the 32 bit words returned for each detector channel:

tesdatar00c00
tesdatar00c01
...

Note that MAS will determine exactly what channels are returning data and name the fields appropriately. So when reading RC2 only with readout_row_index=4 and num_rows_reported=1, the output fields will be

tesdatar04c08
tesdatar04c09
tesdatar04c10
tesdatar04c11
tesdatar04c12
tesdatar04c13
tesdatar04c14
tesdatar04c15

Derived fields

Based on the Data Mode of each readout card, derived signals are extracted from the raw data and made available to libgetdata users. The field extraction and conversion is accomplished according to the same rules used by mce_data.py and mas_data.pro , i.e. the data are scaled to some reference level for that signal. See the discussion in the Data Mode page.

The field names for error, feedback, filtered feedback, and phi0 counter data are, respectively (for r00c00)

error_r00c00
fb_r00c00
filt_r00c00
fj_r00c00

For some conversions, it is necessary to first extract the bitfield from the raw MCE word and then scale it. In these cases an intermediate field will be created, e.g.:

INTER_filt_r00c00

Such fields can be ignored.

Warning about signed fields in mixed-mode data

Due to a request from the Matt on behalf of the MCE community, a signed bitfield derived type has been added to the Dirfile Standards. It has been available since Standards Version 7 (October 2009), although kst-1 binary packages, and even some GetData binary packages, often still don't support it.

MAS, in turn, has been modified to write by default these signed bitfields (SBIT). In order to support older versions of GetData and kst, however, they may be turned off by using the ACQ_OPTION command in mce_cmd to change the standards version used. If signed bitfields are disabled, the result is that signed signals from mixed-mode MCE data will appear as positive numbers, with 'negative' values -D represented by the large positive number 2^n - D. For example, in data mode 4 the error signal is encoded in the lowest-order 14 bits of the MCE word. An error reading of 12 gets encoded as 00000000001100 in binary and would be extracted as 12, but an error reading of -12 would be encoded with the 14 binary digits 11111111110100, and upon extraction would end up as 2^14 - 12 = 16372.

Sign-related issues can be corrected in down-stream software; there is no information loss. But upgrading your GetData/kst is encouraged to fix this problem.