Community Help

reynlds · ‎01-22-2025

Got this little ditty today:

2025-01-22 10:05:05,294 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: JcMid_eNLgZT_7qI2WBseNq4zJlRhXRajHT5Rlm1AUrGwdO6FP9m2A==

Along with the "malformed HTTP response" errors, I'm receiving these on a regular basis:

- ERROR - table not initialized, use `initdb`
- ERROR: The request could not be satisfied
- ERROR - invalid_client

My target DB is a horribly over-resourced PG instance in AWS/RDS and I'm using DAP client ver. 1.3.1 on Python 3.12.5. These errors have only coming fast 'n furious in the last few weeks. Before that it's been smooth sailing. Thoughts?

reynlds · ‎02-11-2025

Timeouts to the service and other errors continue.

reynlds · ‎02-17-2025

Seeing WAY too many of these lately:

"ERROR - An exception occurred while executing the command: DAP Querying service is overloaded or under maintenance"

What's going on with this service as it doesn't appear to be scaling to meet need. What are are the published maintenance times for DAP??

StevenWalls · ‎02-19-2025

We're getting a bunch of "ERROR - malformed HTTP response" lately. In the past it meant an unusual amount of traffic..

reynlds · ‎02-19-2025

@sgergely Any response to these performance/networking issues we're seeing???

sgergely · ‎02-21-2025

Hello, thank you for the mention @reynlds! Yes, we are seeing unusual amount of traffic and we are working on it. Sorry for the issues it causes.

reynlds · ‎02-21-2025

@sgergely here are summaries of the most recent errors:

025-02-21 06:05:33,902 - ERROR - malformed HTTP response:
2025-02-21 06:05:33,919 - ERROR - malformed HTTP response
2025-02-21 06:05:48,928 - ERROR - malformed HTTP response:
2025-02-21 06:05:48,946 - ERROR - malformed HTTP response
2025-02-21 06:06:07,508 - ERROR - malformed HTTP response:
2025-02-21 06:06:07,526 - ERROR - malformed HTTP response
2025-02-21 06:07:12,645 - ERROR - malformed HTTP response:
2025-02-21 06:07:12,661 - ERROR - malformed HTTP response
2025-02-21 06:07:16,024 - ERROR - malformed HTTP response:
2025-02-21 06:07:16,043 - ERROR - malformed HTTP response
2025-02-21 06:23:09,501 - ERROR - malformed HTTP response:
2025-02-21 06:23:09,519 - ERROR - malformed HTTP response

2025-02-21 00:17:31,592 - ERROR - malformed HTTP response:
<TITLE>ERROR: The request could not be satisfied</TITLE>
<H1>504 ERROR</H1>
2025-02-21 00:17:31,609 - ERROR - malformed HTTP response
2025-02-21 00:25:24,502 - ERROR - malformed HTTP response:
2025-02-21 00:25:24,516 - ERROR - malformed HTTP response
2025-02-21 00:40:15,464 - ERROR - malformed HTTP response:
<TITLE>ERROR: The request could not be satisfied</TITLE>
<H1>504 ERROR</H1>
2025-02-21 00:40:15,481 - ERROR - malformed HTTP response
2025-02-21 00:41:49,074 - ERROR - malformed HTTP response:
<TITLE>ERROR: The request could not be satisfied</TITLE>
<H1>504 ERROR</H1>
2025-02-21 00:41:49,090 - ERROR - malformed HTTP response

sgergely · ‎02-21-2025

Just for me to understand your current situation: can you get any data through CD2 and there are issues time-to-time or nothing comes through because everything fails?

reynlds · ‎02-21-2025

I know exactly which tables are failing from the above error messages. I take that and do a manual syncdb, and the next automated one (which has been in place for years) will fail with a variety of errors. it looks like most of the failures happen overnight, but once a malformed http response happens, it requires manual intervention on the table(s). Having our Db out of sync with CD2 is not good.

reynlds · ‎02-26-2025

Since this last post things have gotten markedly better. However, I can't claim that the service becoming "stable" is due to this post, so I'm not holding my breath that it will stay this way. All we can do is monitor and report, which is what I'll continue doing.

sgergely · ‎03-05-2025

Has it been better already? I have been checking our logs and since last week and we haven't seen any issues. If you have can you please help me with details?

reynlds · ‎03-05-2025

@sgergely Latest errors from this morning. FYI...I only log "errors", not "debug". Also, I wrap the log entries from DAP with some extra to show things like the command used, etc.:

==================================================
=== Processing table: content_participation_counts
= 2025-03-05 06:15:15
= Starting syncdb: content_participation_counts
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table content_participation_counts
2025-03-05 06:15:46,926 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: kfDFvzEfFavJ3FNgRhktP1Mvlfmj-Zt_fs1slgeFY4O5_JsX2_3pEQ==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:15:46,942 - ERROR - malformed HTTP response
==================================================
=== Processing table: course_sections
= 2025-03-05 06:22:12
= Starting syncdb: course_sections
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table course_sections
2025-03-05 06:22:43,528 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: GrM5NnjG_JK44tSb-51TSl6syfly-i8Z0tofY2qUC5ujVRLZ9_YEzw==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:22:43,547 - ERROR - malformed HTTP response
==================================================
=== Processing table: learning_outcome_question_results
= 2025-03-05 06:38:57
= Starting syncdb: learning_outcome_question_results
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table learning_outcome_question_results
2025-03-05 06:39:29,158 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: jOkN0XtqJv4UPjJFabk2N6O9oX5MsF1axYu8YIDI2jRNLWHUhHtSyQ==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:39:29,175 - ERROR - malformed HTTP response
==================================================
=== Processing table: quizzes
= 2025-03-05 06:49:04
= Starting syncdb: quizzes
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table quizzes
2025-03-05 06:49:35,408 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: LBtIiw4p0MuQmX3BEf4imwe_Yn4fPXQkWVHnIpd-mvRQ50vQaIPebA==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:49:35,426 - ERROR - malformed HTTP response

reynlds · ‎03-05-2025

@sgergely my noon run of syncdb on the following 3 tables failed with the exact same errors as above:

assignment_overrides
context_module_progressions
master_courses_master_migrations

Performing a manual syncdb works and moves things forward, but it should never be expected that a subscribing institution should have to do this multiple times a day (or even once a day or week)

More CD2 errors happening lately...

cd2

dap error

Data Services sends incorrect Content-Type when re...

duplicates in Catalog enrollment table snapshot

Historical data of assessment decision when using ...

Roll Call Attendance / Grades Alert

CD2 - how to identify which users / sections have ...

Canvas Data 2 Documentation Down?

CD2: Microsoft Sync State (for MS Teams)

Integrating Canvas with PowerBI?

New Quizzes

Data Services sends incorrect Content-Type when re...

You're signed out

More CD2 errors happening lately...

Community Help

View our top guides and resources: