Running pRUParser: Difference between revisions

From pCT
(Created page with "== Software Function == The pRUParser takes a raw, binary, offload stream of pRU data from a file. It will sort, error check and prunes this in to a raw, binary, stream and s...")
 
No edit summary
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Main Page]] -> [[Control & Readout Software Documentation and Howto's]] -> [[Running pRUParser]]
== Software Function ==
== Software Function ==
The pRUParser takes a raw, binary, offload stream of pRU data from a file.
The pRUParser takes a raw, binary, offload stream of pRU data from a file created by the pDTP Client.
It will sort, error check and prunes this in to a raw, binary, stream  and save this do disk for further prosessing.  
It will sort, error check and prunes this in to a raw, binary, stream  and save this do disk for further prosessing.  
It will lastly print a summary of any errors it detected.
It will lastly print a summary of any errors it detected.
== Protocol Dependencies ==
The pRUParser uses the pRU Data Format Specification v0.2
'''TODO''' add link to protocol document/page


== Usage ==
== Usage ==
Line 15: Line 23:
$./run-pru-parser -i relative/path/to/input/file -o relative/path/to/output/file -v verbosity level (0 or 1)
$./run-pru-parser -i relative/path/to/input/file -o relative/path/to/output/file -v verbosity level (0 or 1)
</syntaxhighlight>
</syntaxhighlight>
== Output ==
During the parse, if verbosity is greater than 0, a status summaries will be printed:
<syntaxhighlight lang="bash">
Starting pipeline ...
Input policy  [File reader]: file readout/testDataX288.bin size kxcd
Filter policy [Forward]: buffer of kxcd byte(s)
This parse had a speed of: kx.cd Mb/s
Ouput policy  [File writer]: write to file readout/pData.bin size kxcd
... done
</syntaxhighlight>
Otherwise, only a the parse benchmark will be printed
<syntaxhighlight lang="bash">
This parse had a speed of: kx.cd Mb/s
</syntaxhighlight>
Then a summary is printed:
<syntaxhighlight lang="bash">
SUMMARY -------------
Total number of pRU frames parsed: n
Total number of pRU empty words found: n
Total number of pRU delimiter words found: n
Unfinished pRU words in buffer: n
</syntaxhighlight>
If any pRU empty words are received a shot summary will be printed:
<syntaxhighlight lang="bash">
EMPTY FRAMES FOUND: n
</syntaxhighlight>
If no errors are found, this will be printed:
<syntaxhighlight lang="bash">
No errors found.
</syntaxhighlight>
Otherwise an error summary will be printed:
<syntaxhighlight lang="bash">
INPUT_SIZE_ERROR: n, MULTIPLE_PRU_HEADER_ERROR: n, DATA_WORD_MISSING_HEADER_ERROR: n,
TRAILER_WORD_MISSING_HEADER_ERROR: n, PRU_TRAILER_FLAGS_ERROR: n, NO_DATA_WORD_ERROR: n,
FRAME_SIZE_ERROR: n, MISSING_CHIP_HEADER_ERROR: n, MISSING_REGION_HEADER_ERROR: n,
MISSING_CHIP_TRAILER_ERROR: n, MULTIPLE_CHIP_HEADER_ERROR: n, ALPIDE_TRAILER_FLAGS_ERROR: n,
PADDING_ERROR: n, UNKNOWN_ALPIDE_WORD_ERROR: n
</syntaxhighlight>
If any of the non fatal alpide trailer flags has been detected, a summary will be printed:
<syntaxhighlight lang="bash">
ALPIDE Busy Transition: n
ALPIDE Strobe Extended: n
</syntaxhighlight>
== Parser specification ==
=== I/O formats ===
=== Decoder ===
[[File:PRU Parser ALPIDE Decoder flow chart.jpeg|x400px]]
=== Handling of format errors ===
* recovery
* fatal errors
* propagation/monitoring
=== Generation of test data ===
Clone into the WP3 repository
<syntaxhighlight lang="bash">
cd wp3/software/simulation
pip3 install -r requirements.txt
python3 test_cases.py
</syntaxhighlight>
cd to the desired test case folder. The different test cases are described in test_cases.py
<syntaxhighlight lang="bash">
xxd -r -p testcase_pruformat.txt outputfilename.bin
</syntaxhighlight>
=== Known Issues ===
* Unable to analyze datastream if idle (FF) is thrown into the datastream, needs masking to handle this. (Testcase 5)
** idle is masked by the RU firmware in normal operation
** need to be handled as optional debug feature
** extended error checking if error occurred
* Mistakenly throws missing region header error when we have a frame consisting of only alpide header+trailer with alpide trailer set to b8, which should be a busy violation.
* Parser reports more pru frame errors than there are frames when busies are thrown in randomly. Reports missing trailer errors when all trailers are present.
* All frames with errors are discarded, should be saved somewhere.
** currently all frames with errors are stored in a temporary internal buffer, needs to be saved to file
** consider an error descriptor struct to describe frames in the raw buffer
* Need more detail on ALPIDE trailer flags, the different errors should be stored along with the frame id somewhere. (Additional fields in .root format)
* Random empty frames with/without compression leads to missing region headers.
* No specific check for fatal or data overrun in ALPIDE trailer, which should be a critical warning, and the frame should be saved.
* Conversion of data to .root format is only started after all the data is read out. This should be parallelized so that the conversion may start once the first pru-trailer is received.
** implement after stable error handling is implemented

Latest revision as of 09:23, 1 September 2020

Main Page -> Control & Readout Software Documentation and Howto's -> Running pRUParser


Software Function

The pRUParser takes a raw, binary, offload stream of pRU data from a file created by the pDTP Client. It will sort, error check and prunes this in to a raw, binary, stream and save this do disk for further prosessing. It will lastly print a summary of any errors it detected.

Protocol Dependencies

The pRUParser uses the pRU Data Format Specification v0.2

TODO add link to protocol document/page

Usage

In the readout/ folder in your build directory: For help run:

$./run-pru-parser -h

To parse a file, run:

$./run-pru-parser -i relative/path/to/input/file -o relative/path/to/output/file -v verbosity level (0 or 1)

Output

During the parse, if verbosity is greater than 0, a status summaries will be printed:

Starting pipeline ... 
Input policy  [File reader]: file readout/testDataX288.bin size kxcd
Filter policy [Forward]: buffer of kxcd byte(s)
This parse had a speed of: kx.cd Mb/s
Ouput policy  [File writer]: write to file readout/pData.bin size kxcd
... done

Otherwise, only a the parse benchmark will be printed

This parse had a speed of: kx.cd Mb/s

Then a summary is printed:

SUMMARY -------------
Total number of pRU frames parsed: n
Total number of pRU empty words found: n
Total number of pRU delimiter words found: n
Unfinished pRU words in buffer: n

If any pRU empty words are received a shot summary will be printed:

EMPTY FRAMES FOUND: n

If no errors are found, this will be printed:

No errors found.

Otherwise an error summary will be printed:

INPUT_SIZE_ERROR: n, MULTIPLE_PRU_HEADER_ERROR: n, DATA_WORD_MISSING_HEADER_ERROR: n, 
TRAILER_WORD_MISSING_HEADER_ERROR: n, PRU_TRAILER_FLAGS_ERROR: n, NO_DATA_WORD_ERROR: n, 
FRAME_SIZE_ERROR: n, MISSING_CHIP_HEADER_ERROR: n, MISSING_REGION_HEADER_ERROR: n, 
MISSING_CHIP_TRAILER_ERROR: n, MULTIPLE_CHIP_HEADER_ERROR: n, ALPIDE_TRAILER_FLAGS_ERROR: n, 
PADDING_ERROR: n, UNKNOWN_ALPIDE_WORD_ERROR: n

If any of the non fatal alpide trailer flags has been detected, a summary will be printed:

ALPIDE Busy Transition: n
ALPIDE Strobe Extended: n

Parser specification

I/O formats

Decoder

PRU Parser ALPIDE Decoder flow chart.jpeg

Handling of format errors

  • recovery
  • fatal errors
  • propagation/monitoring

Generation of test data

Clone into the WP3 repository

cd wp3/software/simulation
pip3 install -r requirements.txt
python3 test_cases.py

cd to the desired test case folder. The different test cases are described in test_cases.py

xxd -r -p testcase_pruformat.txt outputfilename.bin


Known Issues

  • Unable to analyze datastream if idle (FF) is thrown into the datastream, needs masking to handle this. (Testcase 5)
    • idle is masked by the RU firmware in normal operation
    • need to be handled as optional debug feature
    • extended error checking if error occurred
  • Mistakenly throws missing region header error when we have a frame consisting of only alpide header+trailer with alpide trailer set to b8, which should be a busy violation.
  • Parser reports more pru frame errors than there are frames when busies are thrown in randomly. Reports missing trailer errors when all trailers are present.
  • All frames with errors are discarded, should be saved somewhere.
    • currently all frames with errors are stored in a temporary internal buffer, needs to be saved to file
    • consider an error descriptor struct to describe frames in the raw buffer
  • Need more detail on ALPIDE trailer flags, the different errors should be stored along with the frame id somewhere. (Additional fields in .root format)
  • Random empty frames with/without compression leads to missing region headers.
  • No specific check for fatal or data overrun in ALPIDE trailer, which should be a critical warning, and the frame should be saved.
  • Conversion of data to .root format is only started after all the data is read out. This should be parallelized so that the conversion may start once the first pru-trailer is received.
    • implement after stable error handling is implemented