DAG: nd201905_kpi

schedule: 0 7,15 * * *


Task Instance: getCNCEData


Task Instance Details

Dependencies Blocking Task From Getting Scheduled
Dependency Reason
Task Instance State Task is in the 'success' state which is not a valid state for execution. The task must be cleared in order to be run.
Dagrun Running Task instance's dagrun was not in the 'running' state but in the state 'success'.
Attribute: python_callable
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
def getCNCEData(**context):
    ti = context["ti"]
    token = ti.xcom_pull(key="token")

    response = requests.get(
    url=f"{dRoW_api_end_url}/api/module/document-export/airflow/workflow/637c7d22b38f8ca02f5c49ab?export_type=0",
    headers={
        "x-access-token": f"Bearer {token}",
        "ICWPxAccessKey": "5WSD21ICWP_[1AG:4UdI){n=b~"
        }
    )

    cnceData = response.json()
    print(f"check cnce data: {cnceData}")

    today = datetime.now(timezone.utc)

    rows = []
    for item in cnceData:
        d = (item.get("data") or {})
        event_no = (d.get("NEC Event No.") or "").strip()
        doc_type = (d.get("NEC Doc Type") or "").strip()
        doc_date_raw = d.get("Doc Date")  # boss wants this
        receive_date_raw = d.get("Receive Date")

        # pick a best-effort "doc_datetime"
        doc_dt = parse_any_dt(doc_date_raw) or parse_any_dt(receive_date_raw)

        if not event_no or not doc_type or not doc_dt:
            continue

        rows.append({
            "event_no": event_no,
            "doc_type": doc_type.strip().upper(),
            "doc_dt": doc_dt,
        })

    # 2) group by event_no and find latest doc
    latest_by_event = {}
    for r in rows:
        key = r["event_no"]
        cur = latest_by_event.get(key)
        if (cur is None) or (r["doc_dt"] > cur["doc_dt"]):
            latest_by_event[key] = r

    # 3) unreplied NCE = latest doc is NCE
    unreplied_nce = []
    for event_no, latest in latest_by_event.items():
        if latest["doc_type"].startswith("NCE"):
            unreplied_nce.append(latest)  # keep latest doc row (its doc_dt is the NCE date)

    # 4) KPI rules (calendar months)
    kpi5_cutoff = today - relativedelta(months=3)
    kpi6_cutoff = today - relativedelta(months=6)

    kpi5 = sum(1 for r in unreplied_nce if r["doc_dt"] < kpi5_cutoff)
    kpi6 = sum(1 for r in unreplied_nce if r["doc_dt"] < kpi6_cutoff)

    print("unreplied NCE total:", len(unreplied_nce))
    print("KPI 5:", kpi5)
    print("KPI 6:", kpi6)

    # push into XCom so writeKpiToPostgres can store
    ti.xcom_push(key="kpi5", value=int(kpi5))
    ti.xcom_push(key="kpi6", value=int(kpi6))

    return {"kpi5": int(kpi5), "kpi6": int(kpi6)}
Task Instance Attributes
Attribute Value
dag_id nd201905_kpi
duration 155.091491
end_date 2025-12-16 15:10:44.635330+00:00
execution_date 2025-12-16T07:00:00+00:00
executor_config {}
generate_command <function TaskInstance.generate_command at 0x7fb2f39d0320>
hostname a7c46ba165e9
is_premature False
job_id 4390
key ('nd201905_kpi', 'getCNCEData', <Pendulum [2025-12-16T07:00:00+00:00]>, 2)
log <Logger airflow.task (INFO)>
log_filepath /usr/local/airflow/logs/nd201905_kpi/getCNCEData/2025-12-16T07:00:00+00:00.log
log_url http://localhost:8080/admin/airflow/log?execution_date=2025-12-16T07%3A00%3A00%2B00%3A00&task_id=getCNCEData&dag_id=nd201905_kpi
logger <Logger airflow.task (INFO)>
mark_success_url http://localhost:8080/success?task_id=getCNCEData&dag_id=nd201905_kpi&execution_date=2025-12-16T07%3A00%3A00%2B00%3A00&upstream=false&downstream=false
max_tries 1
metadata MetaData(bind=None)
next_try_number 2
operator PythonOperator
pid 963352
pool default_pool
prev_attempted_tries 1
previous_execution_date_success 2025-12-16 04:52:25.593744+00:00
previous_start_date_success 2025-12-16 04:55:25.363337+00:00
previous_ti <TaskInstance: nd201905_kpi.getCNCEData 2025-12-16 04:52:25.593744+00:00 [success]>
previous_ti_success <TaskInstance: nd201905_kpi.getCNCEData 2025-12-16 04:52:25.593744+00:00 [success]>
priority_weight 2
queue default
queued_dttm 2025-12-16 15:08:05.432345+00:00
raw False
run_as_user None
start_date 2025-12-16 15:08:09.543839+00:00
state success
task <Task(PythonOperator): getCNCEData>
task_id getCNCEData
test_mode False
try_number 2
unixname airflow
Task Attributes
Attribute Value
dag <DAG: nd201905_kpi>
dag_id nd201905_kpi
depends_on_past False
deps {<TIDep(Trigger Rule)>, <TIDep(Previous Dagrun State)>, <TIDep(Not In Retry Period)>}
do_xcom_push True
downstream_list [<Task(PythonOperator): writeKpiToPostgres>]
downstream_task_ids {'writeKpiToPostgres'}
email None
email_on_failure True
email_on_retry True
end_date None
execution_timeout None
executor_config {}
extra_links []
global_operator_extra_link_dict {}
inlets []
lineage_data None
log <Logger airflow.task.operators (INFO)>
logger <Logger airflow.task.operators (INFO)>
max_retry_delay None
on_failure_callback None
on_retry_callback None
on_success_callback None
op_args []
op_kwargs {}
operator_extra_link_dict {}
operator_extra_links ()
outlets []
owner airflow
params {}
pool default_pool
priority_weight 1
priority_weight_total 2
provide_context True
queue default
resources None
retries 1
retry_delay 0:05:00
retry_exponential_backoff False
run_as_user None
schedule_interval 0 7,15 * * *
shallow_copy_attrs ('python_callable', 'op_kwargs')
sla None
start_date 2023-01-17T00:00:00+00:00
subdag None
task_concurrency None
task_id getCNCEData
task_type PythonOperator
template_ext []
template_fields ('templates_dict', 'op_args', 'op_kwargs')
templates_dict None
trigger_rule all_success
ui_color #ffefeb
ui_fgcolor #000
upstream_list [<Task(PythonOperator): getCEImplementationData>]
upstream_task_ids {'getCEImplementationData'}
wait_for_downstream False
weight_rule downstream