Python data and runfile modules

Recent changes:

• mce_script/trunk r408: Merged mce_runfile.py into mce_data.py. Use mce_data.py for everything.
• mce_script/trunk r407: SmallMCEFile is smarter about frame counts, and less verbose unless challenged.

The interfaces described here are based on mce_script/trunk r408.

mce_data.py and mce_runfile.py

Notes

The module mce_data.py provide roughly the same functionality that mas_data.pro and mas_runfile.pro provide for IDL.

In the latest versions of the scripts, mce_runfile.py is no longer needed as the MCERunfile class has been merged into mce_data.py

System setup

Make sure python can find mce_data.py (and possibly mce_runfile.py). $MAS_PYTHON should be in your PYTHONPATH environment variable.

mce@mce-ubc-2:~$ export PYTHONPATH=$PYTHONPATH:$MAS_PYTHON

This should be done automatically if you "source $MAS_ROOT/template/mas_env.bash" in your .bashrc".

MCEFile / SmallMCEFile classes

The MCEFile class is an alias for the SmallMCEFile class. These can be used interchangeably. The "Small" is just meant to suggest that you shouldn't try to load more frame data than your machine's RAM can handle. The module will complain and prevent you from loading more than 100 MB of data in one go.

Basic usage

In python, import the 'mce_data' module and create a SmallMCEFile object:

>>> from mce_data import *
>>> fn = '/data/cryo/current_data/data002'
>>> f = SmallMCEFile(fn)

To suppress loading of the runfile, or override the runfile name, use one of these:

>>> f = SmallMCEFile( fn, runfile=False)
>>> f = SmallMCEFIle( fn, runfile='template_runfile.run')

Then load the data like this:

>>> d = f.Read()

The data member of d is a 2d array, [n_detectors, n_frames]:

>>> d.data.shape
(328, 100)

The list of rows and columns associated with the 328 detectors is available through col_list and row_list:

>>> d.col_list
[8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
 8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
 ... 
 8, 9, 10, 11, 12, 13, 14, 15]

To display a pixel's values in all the data frames of a file:

>>> d.data[col+row*8]

Or to see pixel index i:

>>> d.data[i,:]

To display all pixels in frame f:

>>> d.data[:,f]

The data from the first frame header is available in the header member:

>>> d.header
{'status': 2052, 'data_rate': 47, 'userfield': 0, 'num_rows_reported': 41,
 'runfile_id': 1231969044, 'row_len': 64, 'header_version': 6, 'rc_present':
 [False, True, False, False], 'address0_ctr': 23142826, 'num_rows': 41,
 'ramp_value': 0, 'frame_counter': 0, 'sync_box_num': 0, 'ramp_addr': 0}

Raw frame data

To get raw frames (including header and checksum), pass "raw_frames=True" to the Read call:

>>> raw = f.Read(raw_frames=True)
>>> raw.shape
(100, 1100)

Then you can look at, e.g., the checksum:

>>> raw[:,1099]
>>> dd[:,1099]
array([918158926, 918144283, 918156148, 918156615, 918153934, 918159205,
       ...
       918146379, 918145339, 918158232, 918146376], dtype=int32)

Force row/column ordering

To reformat the d.data array so that its indices are (row, column, frame), pass "row_col=True" to Read:

>>> d = f.Read(row_col=True)
>>> d.data.shape
(33, 32, 100)

Then to see frame k:

>>> d.data[:,:,k]

Extracting fields from mixed mode data

The MCE signal names, for mce_data purposes, are:

error    - co-added error (reference mode 0)
fb       - feedback in sq1 DAC units (reference mode 1)
fb_filt  - filtered feedback (reference mode 2)
fj       - flux jump counter

For mixed modes, the default is to extract the feedback or filtered feedback signal by default. To extract an alternate signal, pass "field=..." to the Read call:

>>> d = f.Read(field='fj')
>>> print d.data
 array([[0,0,0,0,1,0,0,0,0,
 ...

It is possible to extract a set of fields simultaneously, using the "fields=" option in the Read call. This will override the "field=" option. The extracted fields appear in a dictionary in d.data.

>>> d = f.Read(fields=['fj', 'fb_filt'])
>>> print d.data['fj']
    ...
>>> print d.data['fb_filt']
    ...

You can extract all fields by passing "fields='all'":

>>> d = f.Read(fields='all')
>>> print d.data['fj']
    ...
>>> print d.data['fb_filt']
    ...

To force the bitfield extraction code to assume a particular data_mode, pass "data_mode=..." to Read.

>>> d = f.Read(data_mode=10, field='fb_filt')

MCERunfile class

Note that in older versions of mce_script, the MCERunfile class is contained in the file mce_runfile.py. In that case, replace "from mce_data import ..." with "from mce_runfile import ..." in what follows.

Load a runfile

>>> from mce_runfile import *
>>> runfile_name = '/data/cryo/current_data/1220531790_dat.run'
>>> rf = MCERunfile(runfile_name)

Simple data

Recall that the structure of runfiles is such that a line of data has an address defined by its 'block' and 'key' (where the key is the tag + specifiers...). The contents of rf include a dictionary of dictionaries of all the block / key pairs. For example:

>>> print rf.data['HEADER']['RB rc1 data_mode']
 00000010
>>> print rf.data['SQUID']['SQ_tuning_dir']
 1220510497

However, the member function "Item" allows us to repackage the runfile data by specifying a data type ('string', 'int', 'float') and whether or not we expect an array or a single value. For example:

>>> print rf.Item('HEADER', 'RB rc1 data_mode')
['00000010']
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int')
[10]
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int', array=False)
10
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='float')
[10.0]

Two-dimensional data

Some runfile entries are really 2d arrays, entered row by row. For example, in the 'IV' block there are per-column entries for responsivity:

<IV>
...
<Responsivity(W/DACfb)_C0> 1.92290e-16 1.92119e-16 0.00000 1.93100e-16 1.92978e-16 1.89769e-16 1.91119e-16 ...
<Responsivity(W/DACfb)_C1> 0.00000 1.82838e-16 0.00000 1.84197e-16 1.84822e-16 1.84447e-16 1.83693e-16 ...
<Responsivity(W/DACfb)_C2> 1.89962e-16 1.88649e-16 0.00000 1.85339e-16 1.84462e-16 1.82965e-16 1.84045e-16 ...
...
</IV>

These can be extracted at once if you pass a printf-style format string to the member function Item2d:

>>> a = rf.Item2d('IV', 'Responsivity(W/DACfb)_C%i', type='float')
>>> print len(a)
32
>>> print len(a[0])
33 
>>> print a[2][1]
1.88649e-16

(i.e. a[2][1] is the responsivity for column 2, row 1.)

Some MCE data are spread across readout cards... e.g. adc_offset is stored in rc# adc_offset#. To accumulate all of these into a single array, use Item2dRC, passing the format in a way such that it can be evaluated first with the RC number and then with the column index... :

>>> a = rf.Item2dRC('HEADER', 'RB rc%i adc_offset%%i', type='int')
>>> len(a)
32
>>> len(a[0])
41