web_logs table not available in GET List of tables

[GreyChristensen]

Hello, I am attempting to warehouse the web_logs table from CD2. I see it in the documentation, however I don't see it when I pull a list of available tables using GEThttps://api-gateway.instructure.com/dap/query/canvas/table. the following is the list I see. I tried to also individually query web_logs, it says the table is not recognized. How can I get our web_logs using CD2?

"access_tokens",

"account_users",

"accounts",

"assessment_question_banks",

"assessment_questions",

"assignment_groups",

"assignment_override_students",

"assignment_overrides",

"assignments",

"attachment_associations",

"attachments",

"calendar_events",

"canvadocs_annotation_contexts",

"comment_bank_items",

"communication_channels",

"content_migrations",

"content_participation_counts",

"content_participations",

"content_shares",

"content_tags",

"context_external_tools",

"context_module_progressions",

"context_modules",

"conversation_message_participants",

"conversation_messages",

"conversation_participants",

"conversations",

"course_account_associations",

"course_sections",

"courses",

"custom_gradebook_column_data",

"custom_gradebook_columns",

"developer_key_account_bindings",

"developer_keys",

"discussion_entries",

"discussion_entry_participants",

"discussion_topic_participants",

"discussion_topics",

"enrollment_dates_overrides",

"enrollment_states",

"enrollment_terms",

"enrollments",

"favorites",

"folders",

"grading_period_groups",

"grading_periods",

"grading_standards",

"group_categories",

"group_memberships",

"groups",

"late_policies",

"learning_outcome_groups",

"learning_outcome_question_results",

"learning_outcome_results",

"learning_outcomes",

"lti_line_items",

"lti_resource_links",

"lti_results",

"master_courses_child_content_tags",

"master_courses_child_subscriptions",

"master_courses_master_content_tags",

"master_courses_master_migrations",

"master_courses_master_templates",

"master_courses_migration_results",

"originality_reports",

"outcome_proficiencies",

"outcome_proficiency_ratings",

"post_policies",

"pseudonyms",

"quiz_groups",

"quiz_questions",

"quiz_submissions",

"quizzes",

"role_overrides",

"roles",

"rubric_assessments",

"rubric_associations",

"rubrics",

"score_statistics",

"scores",

"submission_comments",

"submission_versions",

"submissions",

"user_account_associations",

"user_notes",

"users",

"web_conference_participants",

"web_conferences",

"wiki_pages",

"wikis"

[KeithSmith_au · Accepted solution]

Web logs are a different space to tables.

Try GEThttps://api-gateway.instructure.com/dap/query/canvas_logs/table

(and use the canvas_logs rather than just canvas in the other queries as well)

[m_fowler]

Great! I was just about to ask this exact question. The documentation isn't very clear.  

[reynlds]

Mine's never worked at all, timing out each time I attempt an initdb. I've got a case open on it (case # 10287191). Here is the latest comment from Instructure on it:

"Currently to resolve this issue it is requiring an overhaul to our CLI tool. Currently our Engineering team are working on this overhaul but we do not have a timeline for when this will be released at this time."

I've seen some able to get the canvas_logs.web_logs and I'd like to know how it's happening for them.

[saustin]

Thank you for putting in the case number, @reynlds  We will be following this case as we work through all instructions, feedback, and help to ensure we are following proper processes in order to get web_logs to work.  

Thank you again,

Sharon Austin

[KeithSmith_au]

By not using the CLI.

If the API calls are used directly, and timeouts and key refresh handled, as well as fetching all the resultant data sets, there is no issue fetching the data.

The CLI tool may be convenient, but it is not really something that is compatible with a true Enterprise grade installation - any database schema changes and the like really need to be managed via appropriate change and release control processes, with associated testing.  All of this comes with considerable overhead, but can be considerably more robust (and tailored with respect to control).

That said, whilst fetching web_logs is quite possible (I have no issues doing so when required), there is far too much data for us to be bothered - and we don't really want to store it.

[dtod]

If using the dap client, it's done using the --namespace parameter, e.g. 

dap schema --table web_logs --namespace canvas_logs

[reynlds]

Good luck...still broken as of today:

dap.integration.database_errors.NonExistingTableError: table `web_logs` does not exist in schema `canvas_logs`

 ...even though I can:

$ dap list --namespace canvas_logs
user_agents
web_logs

[J-J-Jason]

I would love to see a way to list the namespaces. Then we could loop through the tables in each namespace and never miss anything.

[reynlds]

select schema_name
from information_schema.schemata;

 Output:

('pg_catalog',)
('information_schema',)
('public',)
('canvas',)
('canvas_logs',)

You won't use everything, but you can export anything that is (or starts with) "canvas". After that you can get a table list based on each individual namespace. Probably overkill though since Instructure tells us specifically that only "canvas" and "canvas_logs" will be in play.

[J-J-Jason]

Thanks! I was thinking more of a list of namespaces from the API, though. Just like you can hit the endpoint /dap/query/canvas/table it would be cool if there was one like /dap/query/canvas/namespace

Jason

[saustin]

This conversation is much appreciated, I have been trying to access web_logs without success.  When using the query above,  I don't get canvas_logs in my output (screenshot attached) -- would you have any insights on this?  Thank you so much!

[J-J-Jason]

You don't have the "canvas" or the "canvas_logs" schema. Have you been able to successfully retrieve any of the cd2 tables?

Jason

[saustin]

Good afternoon @J-J-Jason and thank you so much for that insight!    I saw your response, and immediately tried to find "any" query examples that would return information from "any" valid tables.   It took me a few days to do this, so please accept my apologies for the delay in response.  Yes, I finally was able to extract information from "enrollments".

That said, I don't believe we are fully set up yet.  To that end, I (and our team) will continue to set up for Canvas Data 2, especially as outlined here:  https://community.canvaslms.com/t5/Admin-Guide/What-is-Canvas-Data-2/ta-p/560956#canvas-data-2

I'll continue to work on this, but if you have any further recommended help-pages for information access the "canvas" or "canvas_logs" schema, it would be greatly appreciated.

Thank you SO much!

Sharon Austin

[saustin]

Good morning.   We have found out that we do not have web_logs because we subscribed to the service: Managed Teaching and Learning.   We found that to have access to web_logs, we would need to switch to Managed Canvas Data offering and there would be a need to switch the database from a postgres database to a Redshift instance.