Comprehensive monitoring and troubleshooting for Debit Card file processing
Purpose: This guide provides systematic monitoring and troubleshooting steps for Debit Card file processing. The process involves multiple checkpoints throughout the day to ensure proper file upload, processing, and confirmation.
Key Monitoring Times: 4:00 PM (file upload), 8:00 PM (verification), 8:45 PM (processing confirmation), and next morning (final confirmation). Each checkpoint has specific actions and escalation procedures.
Objective: Bank team initiates the process by uploading subject files to the system starting from 4:00 PM.
Note: No action required from support team at this stage. This is informational to understand the process flow.
Objective: Verify whether files were successfully uploaded by checking the Slack channel #card-order-monitoring at 8:00 PM.
β Success Scenario: If files show green tick with "Received" status, proceed to monitor Step 3 at 8:45 PM.
β Failure Scenario: If files show red cross or "not received" status, immediately proceed to troubleshooting scenarios below.
Check the following before escalating:
Usage: Testing and development environment. Safe for experimental commands and configuration changes.
β οΈ Caution: Production environment. Make changes carefully and always backup configurations before modifications.
# Check if any xlsx files were received today
pods | grep debit
kubectl logs (pod-name)
Action: If no file is received and no errors found, send email to bank team using the template in the Email Templates section.
When email server returns errors:
# Check ext-mail POD for errors
pods | grep mail
kubectl logs (pod-name)
Action: Copy the exact error message from logs and include in email to bank team about Email Server Error.
When mount point fails:
Action: If mount command returns errors, copy the exact error message and include in email to bank team about Sybrin COS Mount Point issue.
When next day alert shows no Device Production Report:
Action: Send email to PineLabs team and copy bank team for the respective country using the template provided.
Objective: Once files are processed at 8:45 PM, verify that another Slack alert is triggered with processing confirmation.
β Success Scenario: If status shows "Ordered" with count, processing is successful. Proceed to morning confirmation.
β Failure Scenario: If no alert or failure status, check upload pod logs for xlsx files and account numbers with date timestamp. Ask bank team to verify and send account number for clearance from MSB.
Objective: Verify the confirmation email received next morning with subject line format: "KE MasterCard Debit Card Report File Downloaded: [Timestamp]"
Subject: KE MasterCard Debit Card Report File Downloaded: Fri Jul 11 08:00:07 EAT 2025
Content: [Country name] [Timestamps] [File processing details]
Cross-verification Required: Compare the morning email details with the previous day's Slack alerts to ensure consistency.
If count is 0 for any country:
debit_card_monitoringordered_timestamp is set for yesterday's date# For Kenya - check from 8:00 AM current date
# Extract card pack ID from logs
kubectl logs upload-pod-name | grep -i "card.pack" | grep "$(date '+%Y-%m-%d')"
# Check in database table
SELECT card_pack_id, ordered_timestamp, status
FROM debit_card_monitoring
WHERE DATE(ordered_timestamp) = CURDATE() - INTERVAL 1 DAY
AND country = 'KE';
Missing records: If records is missing for other dates, send email to PineLabs team using the provided template.
After pinelabs placing file: 1. check file in /DOWNLOAD/ISVC03 and check if it has records or is it empty , 2. If Pinelabs team process the file and places the file in sftp server , check for the respective country in path /DOWNLOAD/ISVC03/ , if file is placed and is in any other directory rather than mentioned above , mail pinelabs to place device production report file in above mentioned path
CC Recipients for all emails:
Escalation Timeline: