Processing AO-40 Data at the Symbol level by James Miller G3RUH 2003 Jun 25 These notes describe methods of processing the 400 symbols/sec generated by an ADC attached to the integrate-and-dump of a G3RUH P3 400 bps PSK Decoder. Symbols could also be generated by a computer's soundcard DSP process, or indeed by simulation. Familiarity with the AMSAT P3 telemetry idiom is assumed. Some sample code is given. For details of the AO-40 satellite telemetry see: http://www.amsat-dl.org/p3d/pubtelem.zip For details of the experimental FEC telemetry encoding format see: http://www.ka9q.net/ao40/ Outline ------- The tasks of the service program are: 1. Service the RS232 port 2. Condition the data to signed format and scale 3. Differentially decode successive symbols 4. Store the data stream in a circular buffer 5. Scan the data for Sync vector/Output frame of data. Analogue to Digital Converter - note ------------------------------------ The 8-bit ADC measures the voltage at the PSK decoder integrate-and-dump at a rate of 400/s. It outputs its byte RS-232 style at 9600 baud. Each of these bytes represents a "symbol". The bytes represent: ADC Symbol Value ------------------------------ Strongest "1" 0xFF Weak 0/1 (erasure) 0x80 (*) Strongest "0" 0x00 ------------------------------ * Note: This level should be 0x80, but is not guaranteed to be so, as it depends on the user installing the ADC with the correct voltage divider resistors, and on the equipment's PSU voltage. In principle things can be calibrated out by software. 1. Service the RS232 port ---------------------- The RS-232 port should be set to 9600 baud, format 8N1 or 8N2. Bytes will arrive at 400/s, so this procedure, and all subsequent processing needs to be efficient. 2. Condition the data to signed format and scale --------------------------------------------- Denoting the byte value read as v%, process thus: v = v%-adc_mean v = v*adc_scale The NOMINAL value of adc_mean is 128. The value is constant, though in a specific installation it may have to be established by a calibration procedure t.b.d. and perhaps retained in a set-up file. The NOMINAL value of adc_scale is 1.0 but could well be higher or lower depending on the audio level the user inputs to the Demodulator hardware. It is a constant that may have to be set up by a calibration procedure t.b.d. and perhaps retained in a set-up file. Note that v and adc_scale are numbers with finer than integer resolution, and will need scaling accordingly if you do not wish to use floating point maths. At this stage, on full amplitude signals, the data should be a number within the range approx -120.0 +120.0 and should not exceed +/-127. It doesn't matter if this range, with good signals, is as little as (say) +/-80. 3. Differentially decode successive symbols ---------------------------------------- The raw data was differentially encoded at the spacecraft; a "1" is represented by a change in channel polarity, and a "0" by no change. Thus, perform: sym= -(v*vo ) /* differential decode ( note "-" sign) */ symbol%=sym/128 /* rescale */ vo=v /* preserve v */ The last operation saves the current value of v for use next time around. It should be initialised to 0. The rescale operation is signed, and may be implemented by an arithmetic shift. symbol% is a signed byte. 4. Store the data stream in a circular buffer ------------------------------------------ An 8192 byte circular buffer is suggested, initialised throughout to 0x80. The pointer should be initialised to 0. symbol% should be converted to unsigned and stored in the buffer: ip_buf[ptr] = symbol%+128 Increment the pointer AFTER the next stage. 5. Scan the data for Sync vector/Output frame of data -------------------------------------------------- The structure of this stage depends on whether we are dealing with FEC data frames or normal uncoded frames. This is because the FEC frames have an embedded sync vector, whereas normal uncoded frames have a preceding sync vector. FEC Frames (5200 symbols) ------------------------- Each time a new symbol has been stored, look back through the preceding 5200 symbols and perform a correlation operation with every 80th symbol against the local sync vector replica. If the sync gain exceeds a threshold, declare sync_65 found. Output the 5200 symbol frame (to the Viterbi/RS decoder package) Normal Uncoded frames (4096 symbols + 16 crcc) ---------------------------------------------- Each time a new symbol has been stored, look back through the preceding 31 symbols and perform a correlation operation with the 31 symbols against the local sync vector replica (PN31). If the sync gain exceeds a threshold, declare sync_31 found. When 514*8 further symbols have been read, the frame has been acquired. Slice the symbols to top bit only, pack them 8 to a byte, do a crcc check, mark the MSB of byte 0 and output the 512 byte block. Examples of both these procedures is appended. They can easily be combined into one package. Sync Detection - note --------------------- Sync detection is performed by correlating symbols against a local replica. The basic operation is: total = sum{ s(i)*S(i) } where sum{} is the sum of the product indicated for i=1 .. N where N = 31 or 65 s(i) are the N (signed) symbols being tested S(i) are the N (signed) local sync vector replica elements; valued +1 or -1 The total must be normalised, and for this we require an estimate of the magnitude of the signal symbols s(): mag = sqrt(sum{ s(i)*s(i)/N }) where sqrt is the square root operation. Dividing these, we have a measure of the process gain, thus: sum{ s(i)*S(i) } sync_gain = ----------------------- sqrt(sum{ s(i)*s(i)/N }) If the symbols represent noise, or data uncorrelated with the sync vector, sync_gain will be a random variable with zero mean. However, if the symbols are the sync vector, their energy will combine coherently and the sync_gain will take a value of order +N. By choosing a threshold above the highest expected values generated due to noise etc [about 5*sqrt(N)], the presence of a sync vector can be reliably detected. Example Coding -------------- The language is BBCBASIC Variables are either: 4-byte integers e.g. ptr% floating point variables, e.g. adc_mean strings e.g. sync$ Arrays: DIM sync% 65 defines a byte array named sync% of 65 bytes Array access: array?i returns the contents of array[i] array?i=x writes byte x to array[i] Hex numbers are written e.g. &12EF Procedures are introduced with DEF PROCxxxx and closed with ENDPROC. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Example of FEC Data collection ============================== DIM ip_buf% 8192 : REM Circular buffer for input stream DIM op_buf% 5200 : REM Output buffer for frame of symbols : PROCinit_RS423 : REM Set up PN65 sync vector bit pattern DIM sync% 65 sync$="11111110000111011110010110010010000001000100110001011101011011000" FOR i%=1 TO 65: b$=MID$(sync$,i%,1): sync%?(i%-1)=VAL(b$): NEXT : REM Q&D : ip_ptr%=0 : REM Circular input buffer pointer vo=0 : REM Initialise "old" i/p value sync_found%=0 : REM gets set to -1 when sync detected. adc_mean =128 : REM Note: need a method to get these two real-time calibrated adc_scale=1.5 : REM to get raw data to have decent span, say 0x10 to 0xF0 : FOR i%=0 TO 8191: ip_buf%?i%=&80: NEXT : REM flush input buffer : REM Main loop; runs forever REPEAT REPEAT v%=FNserial(1,port%,0,0) : REM /very/ crude handler! UNTIL v%<>-1 : REM Waiting for a byte from ADC on serial port : REM new symbol at last ... remember, 400 calls/sec at this point v = v%-adc_mean : REM it's a byte, and "zero" is ~128 v = v*adc_scale : REM rescale range to +/- max sym%=-(v*vo )>>7 : REM Diff decode vo = v : REM Preserve old raw voltage ip_buf%?ip_ptr%=sym%+128 : REM New sym to buffer; REM it's a byte; 0 to 127 = "0" REM 128 to 255 = "1"; ~128 is "poor" PROCtest_sync65(ip_ptr%) IF sync_found% THEN REM ... we've collected 5200 symbols for the FEC block, so save to file. PROCdump_block_5200 sync_found%=0 ENDIF ip_ptr%+=1 : REM update circular ip buffer pointer mod 8192 ip_ptr%=ip_ptr% AND &1FFF UNTIL 0 : REM Forever END REM------------------------------------ REM. SUB-ROUTINES for main loop REM------------------------------------ : DEF PROCtest_sync65(ip_ptr%) sync_volts =0 frame_energy=1 : REM Mustn't be zero or we're open to a divide-by-zero REM error on "silence" ;-) : ptr%=(ip_ptr%-5200) AND &1FFF: REM Start search here FOR i%=0 TO 64 sync_bit%=sync%?i% sym%=ip_buf%?ptr% - 128 : REM -128 to -1 = "0"; 0 to 127 = "1"; ~0 is "poor" IF sync_bit% THEN sync_volts+=sym% ELSE sync_volts-=sym% frame_energy+=sym%*sym% ptr%=(ptr%+80) AND &1FFF : REM Pointer to next sync bit position, MOD 8192 NEXT sync_gain=sync_volts/SQR(frame_energy/65) IF sync_gain>41 THEN sync_found%=-1 ENDPROC : REM Looks back over previous 5200 symbols and see whether sync vector is REM present. REM Max sync_gain = 65 on perfectly uniform, noise free signal REM REM On noise only, the spread of sync_gain 'seen' is about +/-30 (~4 sigma), REM although 35.5 will be seen if you wait long enough REM On noise, likelihood of sync_gain > 40 is remote (~1/day) REM REM On-air experiments show that if sync_gain < 41 is detected, it is REM extremely unlikely that the associated block will FEC decode. REM REM So set threshold at 41. [Evaluated 2003 Jun 19] REM ---------------------------------------------------------------------------- : DEF PROCdump_block_5200 FOR i%=0 TO 5200-1 ptr%=(ip_ptr%-5200+i%) AND &1FFF sym%=ip_buf%?ptr% : REM get symbol from the symbol buffer op_buf%?i%=sym% : REM Copy the symbol to the o/p buffer NEXT REM At this point the FEC block is at op_buf% ready for the Viterbi/RS REM processes, though here we simply dump it to disc. : fh%=OPENOUT "feclog" PTR#fh%=EXT#fh% SYS "OS_GBPB",2,fh%,op_buf%,5200 : REM Append latest block CLOSE#fh% ENDPROC +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Example of Normal Uncoded frame Data collection =============================================== DIM ip_buf% 8192 : REM Circular buffer for input stream DIM op_buf% 514*8 : REM Output buffer for block (bits) DIM op_blk% 514 : REM Output buffer for block (bytes) : PROCinit_RS423 REM Set up PN31 sync vector (0x3915ED30) bit pattern DIM sync% 32: sync$="0011100100010101111011010011000" FOR i%=1 TO 31: b$=MID$(sync$,i%,1): sync%?(i%-1)=VAL(b$): NEXT : ip_ptr%=0 : REM Circular input buffer pointer vo=0 : REM Initialise "old" i/p value sync_found%=0 : REM gets set to -1 when sync detected. adc_mean =128 : REM Note: need a method to get these two real-time calibrated adc_scale=1.5 : REM to get raw data to have decent span, say 0x10 to 0xF0 : FOR i%=0 TO 8191: ip_buf%?i%=&80: NEXT : REM flush input buffer : REM Main loop; runs forever REM Main loop; runs forever REPEAT REPEAT v%=FNserial(1,port%,0,0) : REM /very/ crude handler! UNTIL v%<>-1 : REM Waiting for a byte from ADC on serial port : REM new symbol at last ... remember, 400 calls/sec at this point v = v%-adc_mean : REM it's a byte, and "zero" is ~128 v = v*adc_scale : REM rescale range to +/- max sym%=-(v*vo )>>7 : REM Diff decode vo = v : REM Preserve old raw voltage ip_buf%?ip_ptr%=sym%+128 : REM New sym to buffer; REM it's a byte; 0 to 127 = "0" REM 128 to 255 = "1"; ~128 is "poor" IF sync_found% THEN REM ... we're collecting bits for the block op_buf%?block_bit_count%=sym%+128 block_bit_count%+=1 IF block_bit_count%=514*8 THEN PROCdump_block_512: sync_found%=0 ELSE REM ... we're waiting for sync vector of new block PROCtest_sync31(ip_ptr%) block_bit_count%=0 ENDIF ip_ptr%+=1 : REM update circular ip buffer pointer mod 8192 ip_ptr%=ip_ptr% AND &1FFF UNTIL 0 : REM Forever END REM------------------------------------ REM. SUB-ROUTINES for main loop REM------------------------------------ : DEF PROCtest_sync31(ip_ptr%) sync_volts=0 frame_energy=1 : REM Mustn't be zero or we're open to a divide-by-zero REM error on "silence" ;-) : ptr%=(ip_ptr% - 31) AND &1FFF: REM Start search here FOR i%=0 TO 30 sync_bit%=sync%?i% sym%=ip_buf%?ptr% - 128 : REM -128 to -1 = "0"; 0-127 = "1"; ~0 is "poor" IF sync_bit% THEN sync_volts+=sym% ELSE sync_volts-=sym% frame_energy+=sym%*sym% ptr%=(ptr%+ 1) AND &1FFF : REM Pointer to next sync bit position, MOD 8192 NEXT sync_gain=sync_volts/SQR(frame_energy/31) IF sync_gain>25 THEN sync_found%=TRUE ENDPROC REM Look at latest 31 bits and see whether sync vector is present. REM Max sync_gain = 31 on perfectly uniform, noise free signal REM On noise only, the spread of sync_gain is typ. +/-19 , ~3.4 sigma level REM REM On signal, if sync_gain < 29, crcc OK is very unlikely REM REM Threshold of about 25 ensures some poor blocks for Pauls' bit-merge system REM ---------------------------------------------------------------------------- DEF PROCdump_block_512 crcc%=&FFFF FOR i%=0 TO 513*8 STEP 8 byte%=0 FOR j%=0 TO 7 sym%=op_buf%?(i%+j%) : REM get sym from the sym buffer bit%=(sym%AND&80)>>7 : REM Use MSbit only (Viterbi would use full value) byte%=(byte%<<1) OR bit% : REM Build up the byte PROCbit_crcc : REM Cycle crc division using bit NEXT op_blk%?(i%>>3)=byte% : REM Plant the new byte in o/p block NEXT IF (crcc% AND &FFFF) THEN op_blk%?0=op_blk%?0 OR &80 : REM crc failed, set MSB : REM At this point new block is at op_blk% and is ready for telemetry display REM program. Here, we'll just dump it to disc for now. : fh%=OPENUP "tlmlog" PTR#fh%=EXT#fh% SYS"OS_GBPB",2,fh%,op_blk%,512 : REM Append latest block CLOSE#fh% ENDPROC REM ---------------------------------------------------------------------------- DEF PROCbit_crcc test% = (bit%<<15 EOR crcc%) AND &8000 crcc% = crcc% << 1 IF test% THEN crcc% = crcc% EOR &1021 ENDPROC