LNO Dayside Nadir

 

The LNO dataset has been reprocessed to add a reflectance factor calibration to all files and to improve the spectral calibration. The LNO dayside nadir files now have two observation types, "DP" and "DF", which stand for "dayside nadir pass" and "dayside nadir fail". For example:

  • 20180421_005020_1p0a_LNO_1_DP_169.h5
  • 20180421_005020_1p0a_LNO_1_DF_164.h5

 

 "Pass" indicates that both the diffraction order contains solar lines (to fit the solar reference spectra) and the nadir data contains solar and/or atmospheric absorption lines of sufficient depth that the pipeline was able to detect them and correctly shift both the solar and nadir spectral calibrations to match the lines. These files contain the best calibration.

 

"Fail" indicates that either:

  1. the diffraction order does not contain sufficiently deep solar lines to detect and correct the solar reference calibration
  2. the nadir data in the file does not contain any solar and/or atmospheric absorption lines of sufficient depth to detect and correct the solar reference calibration
  3. the nadir data in the file is too noisy to fit any solar or molecular lines.

In these cases, the spectral calibration has been estimated from the temperature of the LNO channel rather than a fit to absorption lines. 

A fail does not necessarily mean that the data is not correct, however more care should be taken when interpreting the data - the spectral calibration is likely to be less accurate and there may be residuals in the data where solar lines have not be correctly divided out during the reflectance factor calculation.

 

 

LNO nadir HDF5 datasets

Below is a description of the new or important datasets in the LNO nadir files:

Dataset Description
Science/YReflectanceFactorOld

The old version of the reflectance factor calibration. The baseline here is curved.

Science/YReflectanceFactor

The new version of the reflectance factor calibration using all solar calibration observations (as of January 2023), with all diffraction orders converted and improved calibration at the edges of the detector.

The spectra are calibrated in reflectance factor, where reflectance factor = pi * normalised nadir counts / normalised solar counts / solar-to-nadir scaling factor / cosine(Geometry/MeanIncidenceAngle).

The baseline here is curved.

Science/YReflectanceFactorFlat Flattened reflectance factor (using the new calibration), where a temperature dependent curve has been fitted to all spectra to reduce the curvature in the spectra. The curve is removed systematically from all spectra, so rare spectral features are retained in the data.
Science/YReflectanceFactorBaseline Removed Flattened reflectance factor, using the method of Guillaume (fit to solar irradiance spectrum and individual baseline correction of each spectrum). Note that wide spectral features may be removed by this flattening technique.
--- ---
Science/YRadiance

The "Y" dataset from previous files. This is the radiance calculated from laboratory blackbody measurements, which uses the AOTF and blaze functions to calculate the full radiance hitting each pixel to derive the counts-to-radiance conversion.

This should be used with caution, because 1) it relies on having correct AOTF and blaze functions; and 2) because the blackbody data used for the calibration is not reliable at high diffraction orders (as the planck function decreases and so the signal becomes very small).

Science/YRadianceSimple

The "YAtWavenumber" dataset from previous files. This is the radiance calibrated from laboratory blackbody measurements without using the AOTF or blaze functions.

The counts to radiance conversion is derived from a simple planck function at the temperature of the blackbody and wavenumber of the pixel. This calibration can also be unreliable for high diffraction orders.

Science/YRadianceError Error in the YRadiance dataset, taken from a fit to the continuum in the first 50 pixels
Science/SNRRadiance Radiance SNR, i.e. YRadiance/YRadianceError
Science/YUnmodified Raw counts unmodified from the level 0.3A HDF5 file
Science/YNormalisedCounts Raw nadir counts, normalised to counts per pixel per second
--- ---
Geometry/MeanIncidenceAngle Mean incidence angle (1 value per spectra). It is the mean of the Point0 - Point5 IncidenceAngle values for that spectrum. 
--- ---
Criteria/LineFit/NumberOfLinesFit Number of molecular/solar lines in the nadir data that were successfully fit
Criteria/LineFit/ChiSqError A number indicating the quality of the solar/molecular line fit (higher means the fit was better)
Criteria/LineFit/Error 0=fit failed, 1=fit passed
--- ---
Channel/MeasurementTemperature Mean LNO baseplate temperature during the observation, taken from the TGO temperature readouts
--- ---
Temperature/TemperatureDateTime Datetimes for the temperature measurements in this dataset group. Temperatures start 10 minutes before the observation begins and end ten minutes after
Temperature/NominalSO Temperature of the SO channel as recorded by TGO. The value is set to -999.0 if no data was recorded
Temperature/NominalLNO Temperature of the LNO channel as recorded by TGO. The value is set to -999.0 if no data was recorded
Temperature/NominalUVIS Temperature of the UVIS channel as recorded by TGO. The value is set to -999.0 if no data was recorded
Temperature/RedundantSO Temperature of the SO channel (redundant sensor) as recorded by TGO. The value is set to -999.0 if no data was recorded
Temperature/RedundantLNO Temperature of the LNO channel (redundant sensor) as recorded by TGO. The value is set to -999.0 if no data was recorded

 

 

UVIS Occultation

 

UVIS grazing occultations have recently been added to the ftp in the hdf5_level_1p0a directory. Each grazing occultation is split into two separate files, one for the ingress (from the top of atmosphere up to the minimum tangent altitude) and one for the egress (up to the top of atmosphere). Hence two filenames, GI and GE respectively are required, for example:

  • 20180508_024553_1p0a_UVIS_GI
  • 20180508_024553_1p0a_UVIS_GE

 

HDF5 datasets

 

Several different algorithms are used to convert the UVIS occultation data to transmittance. The table below lists the dataset names and a description of each.

Science/Y 

Transmittance is computed using a linear extrapolation of the "Sun region" (i.e. where the tangent point is above the atmosphere) for each pixel individually. This method attempts to correct for limb darkening: during a solar occultation, the pointing is not fixed on the same region of the solar disk, and so the signal from the Sun can increase or decrease as the field of view moves towards the centre or edge of the solar disk. This results in a sloped Sun region which is then fit with a linear line of best fit and extrapolated into the "Penumbral region" (i.e. where the tangent point is passing through the atmosphere). The extrapolation is then used to calculate a solar reference spectrum to calibrate each spectrum measured when the line of sight passes through the atmosphere. See Trompet et al. for a more detailed description.

Advantages: the limb darkening effect is reduced by the linear extrapolation

Disadvantages: the extrapolation is done separately for each pixel, therefore noisy pixels could be fit badly, resulting in a poor extrapolation.

Science/YMean

The solar reference spectrum is computed per pixel by taking the mean of all the spectra in the defined "Sun region" (i.e. where the tangent point is above the atmosphere).

Advantages: this method averages together multiple spectra, and so the solar reference spectrum is less noisy and less prone to extrapolation errors due to noisy pixels.

Disadvantages: if the "Sun region" is not flat, e.g. due to limb darkening, the solar reference spectrum used to convert all the atmospheric spectra to transmittance will be too high or low. The transmittance should be 1.00 above the atmosphere, but if limb darkening is not accounted for, the calculated transmittances will be higher or lower than this value.

=> this is the agreed dataset used by the team

Science/YFit <unused> Transmittance computed using a linear regression of the Sun region for each pixel individually. The gradient coefficients (A in Ax+B) are fitted using a 6th order polynomial. This method attempts to reconcile the two methods above: the extrapolation is performed per pixel, but the gradient calculated for each pixel is effective smoothed so that a pixel cannot have a very different gradient to its neighbouring pixels. This aims to correct badly fit extrapolations e.g. those on noisy pixels.
Science/YError Uncertainties on Y
Science/YErrorMean Uncertainties on YMean
Science/YErrorFit <unused> Uncertainties on YFit 
Science/YErrrorNorm <unused> Uncertainties on Y, except that the solar reference spectra are normalised before the standard deviation is made. This dataset is present to remove a known issue in the SO channel calibration.
Science/YErrorMeanRMS M. Wolff error analysis, version 2 (with 2nd derivative constraint)

Science/YErrorMeanRandom

Random errors as propagated through the various stages of the data pipeline

=> this is the agreed dataset used by the team
Science/YErrorMeanSystematic Systematic errors (i.e. straylight) as propagated through the various stages of the data pipeline
Science/YUnmodified Pixel counts before conversion to transmittance

 

 

SO Occultation

HDF5 datasets

 

Several different algorithms are used to convert the SO occultation data to transmittance. The table below lists the dataset names and a description of each.

Science/Y 

Transmittance is computed using a linear extrapolation of the "Sun region" (i.e. where the tangent point is above the atmosphere) for each pixel individually. This method attempts to correct for limb darkening: during a solar occultation, the pointing is not fixed on the same region of the solar disk, and so the signal from the Sun can increase or decrease as the field of view moves towards the centre or edge of the solar disk. This results in a sloped Sun region which is then fit with a linear line of best fit and extrapolated into the "Penumbral region" (i.e. where the tangent point is passing through the atmosphere). The extrapolation is then used to calculate a solar reference spectrum to calibrate each spectrum measured when the line of sight passes through the atmosphere. See Trompet et al. for a more detailed description.

Advantages: the limb darkening effect is reduced by the linear extrapolation

Disadvantages: the extrapolation is done separately for each pixel, therefore noisy pixels could be fit badly, resulting in a poor extrapolation.

=> this is the agreed dataset used by the team

Science/YMean

The solar reference spectrum is computed per pixel by taking the mean of all the spectra in the defined "Sun region" (i.e. where the tangent point is above the atmosphere).

Advantages: this method averages together multiple spectra, and so the solar reference spectrum is less noisy and less prone to extrapolation errors due to noisy pixels.

Disadvantages: if the "Sun region" is not flat, e.g. due to limb darkening, the solar reference spectrum used to convert all the atmospheric spectra to transmittance will be too high or low. The transmittance should be 1.00 above the atmosphere, but if limb darkening is not accounted for, the calculated transmittances will be higher or lower than this value.

Science/YFit Transmittance computed using a linear regression of the Sun region for each pixel individually. The gradient coefficients (A in Ax+B) are fitted using a 6th order polynomial. This method attempts to reconcile the two methods above: the extrapolation is performed per pixel, but the gradient calculated for each pixel is effective smoothed so that a pixel cannot have a very different gradient to its neighbouring pixels. This aims to correct badly fit extrapolations e.g. those on noisy pixels.
Science/YError

Uncertainties on Y, except that the solar reference spectra are normalised before the standard deviation is made. 

=> this is the agreed dataset used by the team

Science/YErrorMean

Uncertainties on YMean

Science/YErrorFit Uncertainties on YFit 
Science/YErrorCovariance Uncertainties calculated using a covariance matrix to remove systematic errors, leaving only the random error. As presented by Adrian and Miguel at SWT#22
Science/CovarianceMatrix The covariance matrix used to calculate the dataset Science/YErrorCovariance

 

 

 

Science/Y example for an SO channel observation

Note that the Sun region is not flat, therefore the extrapolated line has a small negative gradient

 SO transmittance example

 

Science/YFit example for an SO observation

Here the gradient calculated for every pixel (blue) is fit by a polynomial (orange) so that any gradients much larger or smaller than their neighbouring pixels are smoothed. Here the large changes in gradient are caused by solar lines that are shifting slightly due to small changes in the instrument temperature (the spectral calibration is dependent on instrument temperature for SO and LNO). For UVIS this effect should be negigible - however bad gradients due to noisy pixels will be smoothed also.

 

SO yfit example

 

 

SO spectral calibration: updated coefficients in the HDF5 files

 

 Channel/AOTFCentralWavenb  AOTF centre in cm-1 from Villanueva et al. 2022
 Channel/AOTFFunction

 4 parameters from Villanueva et al. 2022:

  •  Sinc width (cm-1)
  •  Sidelobe factor
  •  Asymmetry
  •  Gaussian offset intensity
 Channel/AOTFOrderCoefficients  Coefficients to calculate order from AOTF frequency (unchanged from previous version)
 Channel/AOTFWnCoefficients  Recalculated in data pipeline from new AOTF central wavenumbers
 Channel/SpectralResolution 

 5 parameters from Villanueva et al. 2022:

  •  Intensity of 2nd Gaussian (constant)
  •  Resolving power of both Gaussians (constant)
  •  Displacement of 2nd Gaussian (cm-1) w.r.t. 3700cm-1 vs pixel number (quadratic coefficients)
 Channel/BlazeFunction 

 6 coefficients:

  • Quadratic for FWHM in cm-1 (diffraction order is input variable) - values are unchanged [0, 0, 22.473422]
  • Quadratic for peak position in cm-1 (diffraction order input) - new values [0.0, <FSR>, 0.0] where <FSR> is the temperature + wavenumber corrected value from Villanueva et al. 2022
 Science/X   Wavenumber of each pixel from Trompet et al. 2022  
 Channel/PixelSpectralCoefficients  Pixel and order to wavenumber coefficients from Trompet et al. 2022
 Channel/FirstPixel  Calculated pixel shift due to temperature from Trompet et al. 2022

 

 

SO/LNO/UVIS Solar Calibrations

HDF5 files now pass through the pipeline to level 1.0a

 

HDF5 filenames

Calibration files are assigned additional letter, so miniscans / fullscans / linescans / solar pointings etc. can be distinguished directly from the filename
CM = miniscan
CF = fullscan
CL = linescan

Spectral calibration added to solar fullscans / linescans / pointings (not miniscans)
Y data converted to 2D array to match all other observation files