Why AEP batch ingestion fails: a checklist before you upload

Why this matters

This matters because a failed batch means lost data, delayed insights, and wasted time debugging. Without a proper pre-upload check, errors like missing fields, wrong data types, or invalid formats slip through—especially if only a partial ingestion occurs. Valid records might pass, while failed ones are flagged separately, leading to inconsistent data in the dataset and confusion about what went wrong.

A single malformed record—like "12 kg" in a number field or "yes" in a boolean—can break the entire batch because every record must conform to the schema. Without catching these issues early, the dataset becomes unreliable. Using a small sample to validate against the schema helps avoid costly re-uploads and ensures data integrity before the full batch is processed. This prevents downstream problems in reporting and analysis.

The key ideas

A batch in AEP is uploaded to a single dataset, and each dataset follows one specific schema. That schema defines what data is expected—like required fields, data types, and formats. Every record in the batch must match this schema exactly. If a field is missing, has the wrong type, or doesn't follow the correct format, the record fails. For example, a number field like "age" will fail if it contains "12 kg" or "yes".

The schema enforces strict rules: boolean fields must be true or false, not "yes" or "Y". Date-time fields must use ISO 8601 format, such as 2026-01-15T09:30:00Z. Any source column not mapped to a schema field won’t be included in the batch. An empty file or one with only a header row will produce no records at all. Even if only some records fail, the batch may still partially ingest—valid records go through, while failed ones are reported separately.

Before uploading, always check a small sample of rows against the schema. This catches issues like incorrect data types, wrong formats, or missing fields early. It helps ensure your full batch adheres to the rules of the dataset and avoids failure during ingestion.

How to apply it

Before uploading a batch, verify that every source column is mapped to a schema field. Unmapped columns are ignored entirely, so ensure all data points you intend to include have a defined destination. Next, inspect a small sample of rows—ideally 5 to 10—for schema compliance. Check for missing required fields, incorrect data types, or malformed values. For example, numbers like "12 kg" fail in number fields, and values like "yes" or "Y" don’t pass in boolean fields. Date-time fields must follow ISO 8601 format, such as 2026-01-15T09:30:00Z. Also confirm that no row contains a text value where a number is expected, or vice versa. If the file has only a header or is empty, it will produce no records. Avoid partial ingestion by validating the entire batch structure before upload—partial ingestion may pass some valid records while reporting failures, leading to inconsistent results. Instead, validate the full batch against the schema in a test environment. This step catches most issues early and ensures the dataset ingests cleanly and completely.

Mistakes to avoid

• Always validate data types and formats before uploading. Numbers like "12 kg" fail number fields; ensure values are clean and numeric. Boolean fields must contain exactly "true" or "false"—no "yes", "Y", or other variants. Date-time fields must follow ISO 8601 format, such as 2026-01-15T09:30:00Z. Any deviation causes the record to fail.

• Ensure all source columns are mapped to a schema field. Unmapped columns are ignored entirely. Empty files or files with only headers produce no records. A partial ingestion may pass valid records but report failures, leading to inconsistent results. To catch issues early, check a small sample of rows against the schema before uploading the full batch. This helps identify missing fields, wrong types, or malformed dates before the batch is processed.

Quick checklist

• Verify the file contains actual data rows, not just a header or an empty file
• Confirm all source columns are mapped to a schema field—unmapped columns are not ingested
• Check that required fields are present and correctly typed; missing or malformed fields fail validation
• Ensure numeric fields do not contain text with units (e.g., "12 kg") and that boolean fields use only "true" or "false"
• Validate date-time fields follow ISO 8601 format (e.g., 2026-01-15T09:30:00Z)
• Test a small sample of rows against the schema to catch missing, malformed, or incorrectly typed data
• Review the schema to confirm it matches the expected structure of your data before uploading the full batch