3.3 Application of Compensation Coefficient

To compensate for gain error, each ADC result is multiplied by the calculated coefficient. The compensation can be performed using either fixed-point or floating-point multiplication. The floating-point calculation is shown in ADC Conversion Result Correction Using Floating-Point Calculations and takes 21 instruction cycles. This results in up to 105 ns when the CPU is clocked at 200 MHz.

float coefficient = 3840.0/CalrefH_result; // needed to be done once for each ADC instance
……
// takes 21 instruction cycles or 105 nS @ 200 MHz CPU clock
float corrected_result = (coefficient*((float)AD1CH0DATA));

The multiplication time can be reduced to 6 cycles (30 ns) when fixed-point calculation is used, as shown in 12-bit Unsigned ADC Conversion Result Correction Using Fixed-Point Calculations. Note that the shift left by 18 is to scale up the coefficient to an integer value and minimize rounding errors. The corrected result can be handled in two ways: normalize (right-shift) back to a 12-bit value, or retain the additional LSBs of precision from the larger numerical format. Shifting (truncating) back to 12 bits incurs rounding errors that manifest as a DNL error in the result. To best utilize the improved accuracy of the compensation scheme, it is recommended to retain the additional precision; however, it must be accounted for in its subsequent usage. For simplicity and consistency with floating point, the truncation method is shown in the examples.

int32_t coefficient = (uint32_t) ((1<<18)*3840.0/CalrefH_result; // needed to be done once
// takes 6 instruction cycles or 30 nS @ 200 MHz CPU clock
uint32_t corrected_result = (coefficient*AD1CH0DATA)>>18;