BACKGROUND: I have a scenario where we are having a set of python test script available created using python unittest module. Now ask is to capture logs for the test results in a particular format in some HIVE/impala table.
example:-
target script--> pharma_etl.py performs ETL on source tables and dumps data in to target table.
source table:
- products.X
- products.y
- products.Z
Target table domain.summary_latest
test_pharma_etl.py
consider, we have this script that performs various validations like source table counts match, junk characters check ,etc. and showcase test results in pre-defined format as per unittest module. I see a lot of assert statements in this. we need to reutilize those existing test script and generate output in to a table for further analysis and reporting.
Aucun commentaire:
Enregistrer un commentaire