I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn't uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:
I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
the file is deleted from the google storage bucket.
From my understanding and testing the issue is that job.isDone return true, since the job object is not refresh with the current status.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.
As the code today, there is no flow which the BigQueryException can be raised, so we don't identify ant error in bq load
logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}
I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn't uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:
I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
the file is deleted from the google storage bucket.
From my understanding and testing the issue is that job.isDone return true, since the job object is not refresh with the current status.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.
As the code today, there is no flow which the BigQueryException can be raised, so we don't identify ant error in bq load
logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}