Not all dataFiles get unpacked

[akline1]

The 3 assignment_group files, assignment_rule, and 2 communication_channel files are not being unpacked when I run the sync. Is there an update to the CanvasDataCLI that includes this with the unpack or do I just have to do this manually?

Thanks!

Abe

[kona]

 @akline1 ​, from your question it sounds like this is a Canvas Data question? If so, then the best place to get help and to find useful resources is the Canvas Data​ area of the Community.

Hope this helps!

[akline1]

Thanks, Kona! I was thinking Canvas Data Admin. I'll move it over to the Canvas Data group.

Abe

[James]

 @akline1 ​

This vaguely sounds like Re: canvasDataCli error when unpacking

I can't say for sure that it's the same, but in that case, the downloaded files were corrupted and I showed a way to trick the CLI into re-downloading it.

The CLI doesn't hard-code the names of the tables, it gets it from the schema itself, so there shouldn't have to be a new version to fix those tables. That's what makes me think it's possible a corrupt file issue.

[cward]

 @akline1 ​,

Are you still seeing this issue? If so, please shoot me an email with any additional details around the use case so I can file a ticket with the engineering team. You can email me at cward [at] instructure [dot] com.

The CLI tool is super useful, but its very much a beta product at this stage, so this kind of feedback is awesome.

Chris

--

Product Manager, Data & Analytics

[akline1]

Hey Deactivated user​

Thanks for reaching out. I re-ran the sync this morning and am still missing the same files referenced in my post above in the unpacked folder. Let me know if there is any other info I can provide that would help provide insight to you and the team.

Thanks!

Abe

[cward]

OK, one more question: Are these files showing up in the dataFiles directory when you run the sync before unpacking?

Also, what OS and version of Node.js are you running?

[akline1]

Sure thing, Chris. The files do show up in the dataFiles directory when I

run the sync, so they're just not getting unpacked for some reason. I'm

running Windows 10 and Node.js v5.5.0

Keep the questions coming if there's more info I can provide to assist in

troubleshooting.

Thanks!

Abe

On Mon, Mar 14, 2016 at 5:57 PM, cward@instructure.com <

[cward]

Abe,

I've filed a bug ticket with the team, so it's on their radar. At this point I don't have a time estimate for anyone getting to it, but at least it's been documented.

Chris

[John_Lowe · Accepted solution]

Abe, I just ran into this exact same problem today with these same exact files.  In my case, I was using an unpack command provided here (canvasDataCli unpacking &amp; adding headers ).  I couldn't figure out why these weren't unpacking until I realized that they weren't actually included in the list of files to unpack.  Since then, mine has been working fine.  I hope your situation is as easy to resolve as this.  I sure wish there was a wildcard option with the CLI for this.

This is my new unpack command:

canvasDataCli unpack -c config.js -f account_dim assignment_dim assignment_fact assignment_group_dim assignment_group_fact assignment_group_rule_dim assignment_rule_dim communication_channel_dim communication_channel_fact conversation_dim conversation_message_dim conversation_message_participant_fact course_dim course_section_dim course_ui_navigation_item_dim course_ui_navigation_item_fact discussion_entry_dim discussion_entry_fact discussion_topic_dim discussion_topic_fact enrollment_dim enrollment_fact enrollment_rollup_dim enrollment_term_dim external_tool_activation_dim external_tool_activation_fact group_dim group_fact group_membership_fact pseudonym_dim pseudonym_fact quiz_dim quiz_fact quiz_question_answer_dim quiz_question_answer_fact quiz_question_dim quiz_question_fact quiz_question_group_dim quiz_question_group_fact quiz_submission_dim quiz_submission_fact quiz_submission_historical_dim quiz_submission_historical_fact requests role_dim submission_comment_dim submission_comment_fact submission_comment_participant_dim submission_comment_participant_fact submission_dim submission_fact user_dim wiki_dim wiki_fact wiki_page_dim wiki_page_fact

[cward]

Oy vey, that's bad. Thanks for the update.

[akline1]

Wow, so simple. Just re-ran the unpack with those files added in the command and looks all good. Thanks for posting, John!

[James]

You can future-proof this if you're running on a Linux box.

Assuming that your downloaded dumps are in a subfolder called dataFiles and that you are in the main Canvas Data folder. Then you can run the following from a bash shell:

canvasDataCli -c config.js -f $(find dataFiles -type d | cut -f2 -d/)

or, if you wish to just extract everything quiz-related, you can do this

canvasDataCli -c config.js unpack -f $(find dataFiles -type d -name "quiz_*" | cut -f2 -d/)

[cward]

I've also filed a ticket to ask the engineers to include the ability to use wildcards.

[John_Lowe]

To future proof for Windows users (at least until Instructure engineers allow wildcards):

for /D %f in (dataFiles\*) do canvasDataCli unpack -c config.js -f %~nf

It isn't elegant, but it loops through the directory names in your dataFiles directory and passes that one at a time to the unpack command.  You could probably concatenate that just as easily and only run one unpack, but this gets the job done sequentially.  Don't forget, that if you add this line to a batch file, you much double the %'s like %%f and %%~nf.

[John_Lowe]

I decided that I didn't like my loop for passing one file at a time to the unpack command.  So I've replaced that single line in my script with the following three lines:

set FILELIST=

for /D %%f in (dataFiles\*) do set FILELIST=!FILELIST! %%~nf

call canvasDataCli unpack -c config.js -f%FILELIST%

This change creates a new variable called FILELIST (first line) and concatenates all the directory names in the dataFiles directory into that variable (second line).  Then I just call the unpack command one time (third line) and give it my variable name which contains all directory names created by the earlier SYNC command.  This allows nodejs to do all the multi-threading which greatly increases the speed of the unpack process compared to my first method of sending 56 files one-file-at-a-time to be unpacked with 56 different unpack commands in sequence.

Also, if you are like me and don't want to unpack the huge requests file every time this runs, you can just add the following two lines above and below the snippet above to hide the requests directory during the unpack process and then show it again when finished.

attrib +h /d /s dataFiles\requests

{insert code snippet above here}

attrib -h /d /s dataFiles\requests