mardi 2 février 2021

how to capture custom data validation results / test results for python scripts?

BACKGROUND: I have a scenario where we are having a set of python test script available created using python unittest module. Now ask is to capture logs for the test results in a particular format in some HIVE/impala table.
example:-
target script--> pharma_etl.py performs ETL on source tables and dumps data in to target table.

source table:

  1. products.X
  2. products.y
  3. products.Z

Target table domain.summary_latest

test_pharma_etl.py

consider, we have this script that performs various validations like source table counts match, junk characters check ,etc. and showcase test results in pre-defined format as per unittest module. I see a lot of assert statements in this. we need to reutilize those existing test script and generate output in to a table for further analysis and reporting.

Aucun commentaire:

Enregistrer un commentaire