CD2: Modifying timestamp.py to account for invalid dates

[tyler_clair]

In a response by @ColinMurtaugh in another thread he had mentioned he modified the timestamp.py file of the dap client library to handle ValueErrors thrown by invalid dates. This was an amazing find by Colin and he deserves all the credit. I ran into the same issues and after identifying the offending tables I modified my timestamp.py file to account for these invalid dates and insert a date of 1901-01-01 00:00:00 in place of the invalid date. I was then able to query for this date in those tables and find the resources associated with them and correct them.

Below is my timestamp.py code, I am just sharing it and if you are adventurous and know what you are doing you can modify your client code as well, be aware though, if you update the dap client it will erase the changes made to this file so you will have to change it each time you run a pip upgrade command.

from datetime import datetime, timezone


def valid_utc_datetime(s: str) -> datetime:
    """
    Converts a string into a UTC datetime instance.

    :param s: An ISO 8601 timestamp string.
    :returns: A time zone aware datetime instance with time zone UTC.
    """

    # fromisoformat(...) supports military time zone designator "Zulu" to stand for UTC only in Python 3.11 and later
    try:
        if s.endswith("Z"):
            # remove time zone suffix "Z" (UTC), parse into naive datetime, and explicitly add time zone designator
            return datetime.fromisoformat(s[:-1]).replace(tzinfo=timezone.utc)
        else:
            # parse as time zone aware datetime directly, and convert to UTC
            return datetime.fromisoformat(s).astimezone(timezone.utc)
    except ValueError:
        return datetime(1901, 1, 1, 0, 0, 0, tzinfo=timezone.utc)


def valid_naive_datetime(s: str) -> datetime:
    """
    Converts a string into a naive datetime instance.

    :param s: An ISO 8601 timestamp string.
    :returns: A naive datetime instance that is implicitly assumed to be in time zone UTC.
    """

    # fromisoformat(...) supports military time zone designator "Zulu" to stand for UTC only in Python 3.11 and later
    try:
        if s.endswith("Z"):
            # remove time zone suffix "Z" (UTC) and parse into naive datetime
            return datetime.fromisoformat(s[:-1])
        else:
            # parse as time zone aware datetime, convert to UTC, and cast to naive datetime
            return (
                datetime.fromisoformat(s).astimezone(timezone.utc).replace(tzinfo=None)
            )
    except ValueError:
        return datetime(1901, 1, 1, 0, 0, 0)

[reynlds]

@tyler_clair  Thanks for posting this! I realize that mine isn't working because I had the wrong datetime format in the except clause. Fixed now! Also, thanks to @ColinMurtaugh for identifying this deficiency and pointing toward a workaround.

[Gabor_Endrodi]

@tyler_clair  A new version 0.3.8.1 of the DAP client library has been published to PyPI. It’s more lenient when it’s receiving timestamp values that don’t fully comply with RFC 3339 and ISO 8601. If the timestamp looks like a BCE date, it is replaced with 1 CE (the minimum possible date representable in the four-digit format). If the timestamp looks like a large CE date (e.g. 23000 CE), it is replaced with 9999 CE (the maximum possible date representable in the four-digit format).

You can find the details of each timestamp correction in the logs by searching for "Timestamp format error encountered".

[tyler_clair]

Thanks for letting me know Gabor. This is great update. We cleaned up the original issues with dates but after updating it and having to initialize our tables in a new DB I noticed we had some more, and running the dap client in debug mode reported more errors and hopefully I can fix them in Canvas.