CD2: Trouble with Submissions Table

[asergay]

I can init and sync all the tables except Submissions. It has changed lately, and seems to get farther. I get no error messages, it just stops after downloading 5/11 (is that 5 of 11 objects?). It does not create a Submissions table in postgresql.

Here is the output I'm seeing. Any clues are much appreciated!

2023-08-14 16:29:05,802 - INFO - Query job still in status: running. Checking again in 5 seconds...

2023-08-14 16:29:11,787 - INFO - Data has been successfully retrieved:

{"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4", "status": "complete", "expires_at": "2023-08-15T20:23:10Z", "objects": [{"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00000-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00001-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00002-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00003-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00004-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00005-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00006-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00007-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00008-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00009-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}, {"id": "312070ca-dd91-4a63-b8e0-d143d3a63de4/part-00010-5cd8caf4-2013-471d-9194-fde2fd53c741-c000.json.gz"}], "schema_version": 1, "at": "2023-08-14T17:30:35Z"}

2023-08-14 16:29:13,760 - INFO - Downloading [object 1/11 - job 312070ca-dd91-4a63-b8e0-d143d3a63de4]

2023-08-14 16:29:21,111 - INFO - Downloading [object 2/11 - job 312070ca-dd91-4a63-b8e0-d143d3a63de4]

2023-08-14 16:29:25,554 - INFO - Downloading [object 3/11 - job 312070ca-dd91-4a63-b8e0-d143d3a63de4]

2023-08-14 16:29:27,661 - INFO - Downloading [object 4/11 - job 312070ca-dd91-4a63-b8e0-d143d3a63de4]

2023-08-14 16:30:16,378 - INFO - Downloading [object 5/11 - job 312070ca-dd91-4a63-b8e0-d143d3a63de4]

(base) asergay@H67XP9PQKP ~ % 

Also, every time I see the word submission the Sex Pistols song with that name starts going through my head...

[LeventeHunyadi]

This appears to be a timeout issue while the files are being downloaded and processed. There is likely an error behind the scenes but the error text is not printed.

All SQL insert/update/delete operations are part of a single transaction, you won't see (new) records in the database unless all the files (11 out of 11 in your case) have been fully processed.

I have shared this output with Edina's team. I am sure they will soon circle back.

[asergay]

"I am sure they will soon circle back."

It's been three months. No circle...

[LeventeHunyadi]

The team is currently working on a major revision of the DAP client library, which brings increased speed, improved stability and better extensibility. We are planning to open source the library to the community. We don't have a timeline yet but happy to share when we do. Let me know if you would be interested in testing an initial beta version.

[jcsorenson79]

I'm seeing this same problem with the web_logs table.  Getting very frustrated.  "They" know about it, but haven't heard anything in almost 2 weeks.