Introduction

General introduction

The BlickOperationalProcessingProgressOverview (BOPPO) tries to summarize the processing status, calibration status, and routine data quality control of an instruments dataset.

The underlying gdrive sheet (BOPPO2.0) serves as a workingsheet and has 3 tabs where each is the basis for an automated html:

  • coordination of calibrations of datasets (CalCoordination)
  • documentation/logbook for calibration analysis (BlickLabAnalysisLog)
  • routine quality control (routineQC)

In combination with status information provided by the database, the BOPPO is thought to collect all necessary information about a dataset in an automatized way.

The BOPPO report as presented here focuses on the status of an instrument’s dataset, historic and operational ones. Not all steps are fully implemented at the moment, but it will mainly cover:

  • Which L0 files are available in the database (DB)?
  • What is currently pushed to the PGN server, and since when?
  • What is the current instrument status?
  • Has already someone being assigned to calibrate this dataset, or is it in preparation?
  • Is the dataset being pushed to EVDC, and which gases are released to be uploaded?
  • Is a dataset official PGN?

PGN status

The PGN status is a binary class either being “non-PGN” or “official”. An official PGN status always refers to an instrument plus a certified location. Therefore an instrument is loosing its “official” status if it moves to a temporary location or other location where no certificate has been signed. This also refers to laboratory locations or testing locations. However, if an instrument moves back to a location where it has already been official in the past, it automatically changes it status back to “official”. Therefore, the wording ‘dataset’ is often used and corresponds to a certain PandoraID at a certain location.

EVDC status information

In theory all official PGN instruments are supposed to be visible on EVDC or at least to be public. However, since the focus for satellite validation is on long-term continuous time series, a pre-selection is made which datasets are pushed to EVDC. Therefore, the BOPPO provides an EVDC status subsection which is explicitly overviews datasets that are released.

The corresponding columns explain

  • when the dataset has been released, quality control by the calibration person,
  • which gases are released to be pushed to EVDC,
  • when the last upload/update of a dataset has been made,
  • if a dataset is part of the operational file-push on a daily basis.

Change record

This report is automatically generated 4 times/day and has per definition a dynamic content which is based on the database and google sheets. However, there is a ‘fixed’ structure of sections and information provided by this report as e.g. the overview about the current instruments status. Whenever a new information is added, or different views of how to provide information is changed, this will be part of the change record.

  • 2026-01-28: changing to a new getPGNstatus() function since the home-location is depricated and unused
  • 2024-11-15: bugfix for p1-7 EVDC upload status when the config file is already empty. Removing Datasets per instrument and location,since this view is already in included in the L0-Viewer shiny app
  • 2024-02-20: migrating BOPPO to new server PGN server machine: no EVDC status information about p1-7 given so far
  • 2021-11-18: Added WaitingRoom date to sheet in order to trace the time between WaitingRoom and final assignment to a person
  • 2021-10-12: Added EVDC upload information: date of request, rcodes for gases to be uploaded. Added PGN status information for each dataset.
  • 2021-09-09: Added section ‘Datasets per location’, figure showing instruments at each location
  • 2021-09-01: Added section ‘Datasets per instrument’; figure for locations for each Pandora, and big overview table for available datasets which can be searched better for common locations.
  • 2021-07-17: Changed to interactive tables
  • 2021-05-07: Added section for ‘Unassigned datasets’
  • 2021-05-06: Added EVDC upload status
  • 2021-05-05: First version generated to combine L0 DB information with information from calcoordination

BOPPO

The following subsections are showing only s1 datasets, since there are no routine s2 calibrations done at the Moment. Newly made instruments might not be visible in this report if there is no L0 file pushed to the server so far. Additionally, older datasets being measured with PAN might also not be visible here, which simply means that there are no Blick L0 files being converted.

check if there are any official datasets without L0 files

Current datasets

Current datasets are defined by their last available location a Pandora is submitting L0 files for, if there are multiple locations. Therefore, we can have a current ‘dataset’ for each Pandora where we have L0 files in the database. This also includes testing locations as ElkridgeMD, GreenbeltMD, InnsbruckFKS. And even the famous Aldino, or short campaign locations are counted in the following. However, this does not necessarily mean that there are valid CF’s around, depending on the actual quality and number of measurements taken, which must allow an appropriate calibration. Currently, there are 276 locations given.

Instrument status for current and currently official datasets.
Current datasets Percentage Currrent official datasets Percentage
operational 118 42.8 107 58.2
intensive care 1 0.4 1 0.5
operational with issue 16 5.8 14 7.6
testing 26 9.4 1 0.5
hold due to issue 44 15.9 39 21.2
out of operation 26 9.4 17 9.2
laboratory 30 10.9 4 2.2
maintenance 0 0.0 0 0.0
undefined 15 5.4 1 0.5
total 276 100.0 184 100.0

The following table lists all the current datasets:

Available datasets

The following table lists all available datasets, current one and historic ones.

EVDC status

This section overviews the EVDC upload situation for datasets which have a signed certificate. The following table overviews the datasets which are therefore supposed to be submitted to EVDC.

Currently, 277 datasets are certified. The live-filepush from the PGN server is enabled for 0.

The following table lists datasets which have either been not submitted to EVDC so far, or have obtained an updated calibration which require a new update of an updated bulk-file or daily files, respectively.

Datasets - in progress

Datasets in preparation are defined as having a person assigned to a dedicated datasets (instrument + location). Since it is possible that a dataset has already been calibrated, the person listed always refer to the last person assigned. Therefore, if ‘AssignedDate’ is ‘younger’ than ‘FinishedDate’, a dataset is defined to be in preparation.

Before the final assignment of a calibration team member, the waiting room is to queue needed field calibrations. This is typically the case for field-calibrations needed for new instruments starting to collect L0 files at its first destination, trans-locations, or if an instrument needs a new field-calibration as detected during routineQC. Assigning a dataset to the waiting room simply keeps track of realizing that an action is needed. When there should be enough data and the resources allow it, a member of the calteam is assigned explicitly. The decision tree which instruments to calibrate first is as follows:

  • is the instrument measuring properly?
  • official PGN dataset?
  • dedicated data request?
  • long-term datasets?

Currently there are 30 datasets in preparation by an assigned calibration team member, and 119 are in the waiting room.

Waiting room

The waiting room is to trace datasets which are on the radar to be done for the calibration team, but there are currently no free resources for doing the calibration, or the unit has not enough good days which allow a proper field calibration. This is typically the case if instruments have been detected during routineQC to require a new field calibration, or for new datasets.

In Preparation Per Person

LHaunold

MRoca

IBougoudis

SMorhenn

AKreuter

CWaldauf

MGebetsberger

Datasets - unassigned

Unassigned datasets are the ones, which have not been calibrated so far, which can cover historic datasets, but most importantly new locations. This section is to highlight dataavailability as soon as L0 files are pushed to the server.

Current datasets are easy to filter, since there is typically no person assigned so far. Historic datasets which should be calibrated, are filtered by not having the location of “Aldino”,“ElkridgeMD”,“GreenbeltMD”,“InnsbruckFKS”,“LabGSFC”,“LabSciGlob”,“LabIBK”, and the number of L0 files should at least be 14.
14 is the minimum number of days to do a proper calibration. If the calteam realizes that this minimum number, or even a larger number of e.g., 40 days does not allow a proper calibration, there will be uncalibrated slantcolumns in the CF.

Currently, there are 105 new datasets, and 345 historic datasets being unassigned.

New datasets

Historic datasets