Truncated fields

[phil_mcgachey]

We've been doing some validation across the dataset as a whole, and one thing that has jumped out is that the various free-form text fields are truncated to 255 (or sometimes 256) characters at the schema level. Some of the work that our researchers have done on the edX dataset in the past has involved analyzing forum posts (running tone analysis, for example, and correlating that to outcomes or retention). Obviously that requires the full text of a post, message etc. Would it be possible to add the full body of any free text fields to the dataset, rather than just the first 255 characters?

[rubyn]

We were told that is a Redshift limitation - that we would need to go to the files to get the full text over 256 characters.  I'm not sure if that has changed at all...

[biray]

I believe this is still correct,  @rubyn ​

[rubyn]

Looking into it further:  Character Types - Amazon Redshift

It looks like VARCHAR can hold more than 256 characters.  Is this something that Instructure might change in the future so that entire comments fields will be stored in the database?