web_logs table not available in GET List of tables

[GreyChristensen]

Hello, I am attempting to warehouse the web_logs table from CD2. I see it in the documentation, however I don't see it when I pull a list of available tables using GEThttps://api-gateway.instructure.com/dap/query/canvas/table. the following is the list I see. I tried to also individually query web_logs, it says the table is not recognized. How can I get our web_logs using CD2?

"access_tokens",

"account_users",

"accounts",

"assessment_question_banks",

"assessment_questions",

"assignment_groups",

"assignment_override_students",

"assignment_overrides",

"assignments",

"attachment_associations",

"attachments",

"calendar_events",

"canvadocs_annotation_contexts",

"comment_bank_items",

"communication_channels",

"content_migrations",

"content_participation_counts",

"content_participations",

"content_shares",

"content_tags",

"context_external_tools",

"context_module_progressions",

"context_modules",

"conversation_message_participants",

"conversation_messages",

"conversation_participants",

"conversations",

"course_account_associations",

"course_sections",

"courses",

"custom_gradebook_column_data",

"custom_gradebook_columns",

"developer_key_account_bindings",

"developer_keys",

"discussion_entries",

"discussion_entry_participants",

"discussion_topic_participants",

"discussion_topics",

"enrollment_dates_overrides",

"enrollment_states",

"enrollment_terms",

"enrollments",

"favorites",

"folders",

"grading_period_groups",

"grading_periods",

"grading_standards",

"group_categories",

"group_memberships",

"groups",

"late_policies",

"learning_outcome_groups",

"learning_outcome_question_results",

"learning_outcome_results",

"learning_outcomes",

"lti_line_items",

"lti_resource_links",

"lti_results",

"master_courses_child_content_tags",

"master_courses_child_subscriptions",

"master_courses_master_content_tags",

"master_courses_master_migrations",

"master_courses_master_templates",

"master_courses_migration_results",

"originality_reports",

"outcome_proficiencies",

"outcome_proficiency_ratings",

"post_policies",

"pseudonyms",

"quiz_groups",

"quiz_questions",

"quiz_submissions",

"quizzes",

"role_overrides",

"roles",

"rubric_assessments",

"rubric_associations",

"rubrics",

"score_statistics",

"scores",

"submission_comments",

"submission_versions",

"submissions",

"user_account_associations",

"user_notes",

"users",

"web_conference_participants",

"web_conferences",

"wiki_pages",

"wikis"

[KeithSmith_au · Accepted solution]

Web logs are a different space to tables.

Try GEThttps://api-gateway.instructure.com/dap/query/canvas_logs/table

(and use the canvas_logs rather than just canvas in the other queries as well)

[m_fowler]

Great! I was just about to ask this exact question. The documentation isn't very clear.  

[reynlds]

Mine's never worked at all, timing out each time I attempt an initdb. I've got a case open on it (case # 10287191). Here is the latest comment from Instructure on it:

"Currently to resolve this issue it is requiring an overhaul to our CLI tool. Currently our Engineering team are working on this overhaul but we do not have a timeline for when this will be released at this time."

I've seen some able to get the canvas_logs.web_logs and I'd like to know how it's happening for them.

[saustin]

Thank you for putting in the case number, @reynlds  We will be following this case as we work through all instructions, feedback, and help to ensure we are following proper processes in order to get web_logs to work.  

Thank you again,

Sharon Austin

[KeithSmith_au]

By not using the CLI.

If the API calls are used directly, and timeouts and key refresh handled, as well as fetching all the resultant data sets, there is no issue fetching the data.

The CLI tool may be convenient, but it is not really something that is compatible with a true Enterprise grade installation - any database schema changes and the like really need to be managed via appropriate change and release control processes, with associated testing.  All of this comes with considerable overhead, but can be considerably more robust (and tailored with respect to control).

That said, whilst fetching web_logs is quite possible (I have no issues doing so when required), there is far too much data for us to be bothered - and we don't really want to store it.

[guo848]

Hi, I'm wondering if the web_logs only retain 30 days of data and if there's a way to access the archived logs.

[mclark19]

@guo848 Only 30 days are available via DAP. (https://data-access-platform-api.s3.amazonaws.com/tables/catalog.html#tables-in-canvas-logs) You could contact your CSM to see if they are retained somewhere at Instructure, but I wouldn't hold my breath.

[sgergely]

Hey @guo848 and @mclark19 !

We have the data past 30 days, we don't delete it. The reason we don't publish it is because it is expensive to serve such amount of data and up until now customers were not prepared to get TBs of data. We also have mobile logs as well. Custom development team can help you get the historical log data. You can reach them through your CSM.

Regarding the documentation portal @mclark19 has linked, that is a legacy documentation portal. The new is available here: https://developerdocs.instructure.com/services/dap/dataset/dataset-namespaces/dataset-canvaslogs Please update your bookmarks while I make sure to set redirects to the old documentation pages.

[mclark19]

Thanks @sgergely -- Is there a plan to make mobile log data available via DAP at some point in the future?

[sgergely]

Yes, it is on our roadmap. However I cannot share any timeline for it yet.

[ESheta]

The Web_logs table contains all data, but you can only retrieve up to 30 days of data in a single run. To access historical data, you must use the "since" and "until" parameters, ensuring the time between the two dates does not exceed 30 days. For example, if you need data for one year, you would need to perform multiple queries, each covering a 30-day period. If you need data for a specific period that is less than 30 days, you can simply adjust the "since" and "until" timestamps to retrieve the required data.

[sgergely]

Please note that it is an undocumented DAP Query API feature and can break at any time.

[dtod]

If using the dap client, it's done using the --namespace parameter, e.g. 

dap schema --table web_logs --namespace canvas_logs

[reynlds]

Good luck...still broken as of today:

dap.integration.database_errors.NonExistingTableError: table `web_logs` does not exist in schema `canvas_logs`

 ...even though I can:

$ dap list --namespace canvas_logs
user_agents
web_logs

[J-J-Jason]

I would love to see a way to list the namespaces. Then we could loop through the tables in each namespace and never miss anything.

[reynlds]

select schema_name
from information_schema.schemata;

 Output:

('pg_catalog',)
('information_schema',)
('public',)
('canvas',)
('canvas_logs',)

You won't use everything, but you can export anything that is (or starts with) "canvas". After that you can get a table list based on each individual namespace. Probably overkill though since Instructure tells us specifically that only "canvas" and "canvas_logs" will be in play.

[J-J-Jason]

Thanks! I was thinking more of a list of namespaces from the API, though. Just like you can hit the endpoint /dap/query/canvas/table it would be cool if there was one like /dap/query/canvas/namespace

Jason

[saustin]

This conversation is much appreciated, I have been trying to access web_logs without success.  When using the query above,  I don't get canvas_logs in my output (screenshot attached) -- would you have any insights on this?  Thank you so much!

[J-J-Jason]

You don't have the "canvas" or the "canvas_logs" schema. Have you been able to successfully retrieve any of the cd2 tables?

Jason

[saustin]

Good afternoon @J-J-Jason and thank you so much for that insight!    I saw your response, and immediately tried to find "any" query examples that would return information from "any" valid tables.   It took me a few days to do this, so please accept my apologies for the delay in response.  Yes, I finally was able to extract information from "enrollments".

That said, I don't believe we are fully set up yet.  To that end, I (and our team) will continue to set up for Canvas Data 2, especially as outlined here:  https://community.canvaslms.com/t5/Admin-Guide/What-is-Canvas-Data-2/ta-p/560956#canvas-data-2

I'll continue to work on this, but if you have any further recommended help-pages for information access the "canvas" or "canvas_logs" schema, it would be greatly appreciated.

Thank you SO much!

Sharon Austin

[saustin]

Good morning.   We have found out that we do not have web_logs because we subscribed to the service: Managed Teaching and Learning.   We found that to have access to web_logs, we would need to switch to Managed Canvas Data offering and there would be a need to switch the database from a postgres database to a Redshift instance.