Python data and runfile modules

Python. Way of the future.

The python modules mce_data.py and mce_runfile.py provide roughly the same functionality that mas_data.pro and mas_runfile.pro provide for IDL.

mce_data.py

This provides a way to load SMALL frame data files into python. SMALL means less than 50000 frames.

Basic usage

Make sure python can find mce_data.py and mce_runfile.py. $MAS_PYTHON should be in your PYTHONPATH environment variable.

mce@mce-ubc-2:~$ export PYTHONPATH=$PYTHONPATH:$MAS_PYTHON

In python, import the 'mce_data' module and create a SmallMCEFile object:

>>> from mce_data import *
>>> fn = '/data/cryo/current_data/data002'
>>> f = SmallMCEFile(fn)

To suppress loading of the runfile, or override the runfile name, use one of these:

>>> f = SmallMCEFile( fn, runfile=False)
>>> f = SmallMCEFIle( fn, runfile='template_runfile.run')

Then load the data like this:

>>> d = f.Read()
18600000 items requested but only 37200 read

The data member of d is a 2d array, [n_detectors, n_frames]:

>>> d.data.shape
(328, 100)

The list of rows and columns associated with the 328 detectors is available through col_list and row_list:

>>> d.col_list
[8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
 8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
 ... 
 8, 9, 10, 11, 12, 13, 14, 15]

To display a pixel's values in all the data frames of a file:

>>> d.data[col+row*8]

Or to see pixel index i:

>>> d.data[i,:]

To display all pixels in frame f:

>>> d.data[:,f]

The data from the first frame header is available in the header member:

>>> d.header
{'status': 2052, 'data_rate': 47, 'userfield': 0, 'num_rows_reported': 41,
 'runfile_id': 1231969044, 'row_len': 64, 'header_version': 6, 'rc_present':
 [False, True, False, False], 'address0_ctr': 23142826, 'num_rows': 41,
 'ramp_value': 0, 'frame_counter': 0, 'sync_box_num': 0, 'ramp_addr': 0}

Raw frame data

To get raw frames (including header and checksum), pass "raw_frames=True" to the Read call:

>>> raw = f.Read(raw_frames=True)
55000000 items requested but only 110000 read
>>> raw.shape
(100, 1100)

Then you can look at, e.g., the checksum:

>>> raw[:,1099]
>>> dd[:,1099]
array([918158926, 918144283, 918156148, 918156615, 918153934, 918159205,
       ...
       918146379, 918145339, 918158232, 918146376], dtype=int32)

Force row/column ordering

To reformat the d.data array so that its indices are (row, column, frame), pass "row_col=True" to Read:

>>> d = f.Read(row_col=True)
>>> d.data.shape
(33, 32, 100)

Then to see frame k:

>>> d.data[:,:,k]

Extracting fields from mixed mode data

The MCE signal names, for mce_data purposes, are:

error    - co-added error (reference mode 0)
fb       - feedback in sq1 DAC units (reference mode 1)
fb_filt  - filtered feedback (reference mode 2)
fj       - flux jump counter

For mixed modes, the default is to extract the feedback or filtered feedback signal by default. To extract an alternate signal, pass "field=..." to the Read call:

>>> d = f.Read(field='fj')
>>> print d.data
 array([[0,0,0,0,1,0,0,0,0,
 ...

It is possible to extract a set of fields simultaneously, using the "fields=" option in the Read call. This will override the "field=" option. The extracted fields appear in a dictionary in d.data.

>>> d = f.Read(fields=['fj', 'fb_filt'])
>>> print d.data['fj']
    ...
>>> print d.data['fb_filt']
    ...

You can extract all fields by passing "fields='all'":

>>> d = f.Read(fields='all')
>>> print d.data['fj']
    ...
>>> print d.data['fb_filt']
    ...

To force the bitfield extraction code to assume a particular data_mode, pass "data_mode=..." to Read.

>>> d = f.Read(data_mode=10, field='fb_filt')

mce_runfile.py

Load a runfile:

>>> from mce_runfile import *
>>> runfile_name = '/data/cryo/current_data/1220531790_dat.run'
>>> rf = MCERunfile(runfile_name)

Recall that the structure of runfiles is such that a line of data has an address defined by its 'block' and 'key' (where the key is the tag + specifiers...). The contents of rf include a dictionary of dictionaries of all the block / key pairs. For example:

>>> print rf.data['HEADER']['RB rc1 data_mode']
 00000010
>>> print rf.data['SQUID']['SQ_tuning_dir']
 1220510497

However, the member function "Item" allows us to repackage the runfile data by specifying a data type ('string', 'int', 'float') and whether or not we expect an array or a single value. For example:

>>> print rf.Item('HEADER', 'RB rc1 data_mode')
['00000010']
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int')
[10]
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int', array=False)
10
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='float')
[10.0]

Some runfile entries are really 2d arrays, entered row by row. For example, in the 'IV' block there are per-column entries for responsivity:

<IV>
...
<Responsivity(W/DACfb)_C0> 1.92290e-16 1.92119e-16 0.00000 1.93100e-16 1.92978e-16 1.89769e-16 1.91119e-16 ...
<Responsivity(W/DACfb)_C1> 0.00000 1.82838e-16 0.00000 1.84197e-16 1.84822e-16 1.84447e-16 1.83693e-16 ...
<Responsivity(W/DACfb)_C2> 1.89962e-16 1.88649e-16 0.00000 1.85339e-16 1.84462e-16 1.82965e-16 1.84045e-16 ...
...
</IV>

These can be extracted column by column:

>>> print rf.Item('IV', 'Responsivity(W/DACfb)_C24', type='float')
[1.46858e-16, 1.4528800000000001e-16, 0.0, 1.4240899999999999e-16, 1.4162699999999999e-16, 1.4100399999999999e-16, 1.4088899999999999e-16, 1.39608e-16, 1.3787599999999999e-16, 1.37429e-16, 1.3614499999999999e-16, 1.34423e-16, 1.3543599999999999e-16, 1.3587200000000001e-16, 1.3711399999999999e-16, 0.0, 0.0, 1.39424e-16, 1.40202e-16, 1.4450300000000001e-16, 1.47099e-16, 1.4857100000000001e-16, 1.49762e-16, 1.5139199999999999e-16, 1.5468600000000001e-16, 1.5743600000000001e-16, 1.5934699999999999e-16, 1.62431e-16, 1.6465e-16, 1.65256e-16, 1.6683999999999999e-16, 1.6823900000000001e-16, 0.0]

But they can all be extracted at once if you pass a printf-style format string to the member function Item2d:

>>> a = rf.Item2d('IV', 'Responsivity(W/DACfb)_C%i', type='float')
>>> print len(a)
32
>>> print len(a[0])
33 
>>> print a[2][1]
1.88649e-16

(i.e. a[2][1] is the responsivity for column 2, row 1.)