New feature: Ability for survey respondents to return and modify *completed* responses
In previous versions, once a survey was fully completed, the respondent could not return to make any further edits. (Although users could modify the response on a data entry form if they have appropriate user privileges, but the respondents could not make modifications *on the survey page*.) But now, this new survey option will allow respondents to return after they have fully completed the survey.
This feature can be enabled on the Survey Settings page under the "Save & Return Later" section for a given instrument in the Online Designer. Once enabled, a respondent will be able to return to their response and make any edits to it even if they have fully completed the survey.
As part of the "Save & Return Later" feature, respondents will need to provide a Return Code in order to make edits (as is the case in previous versions when returning to partially completed responses). When the feature is enabled, they will be given their Return Code when they finish the survey, after which they may return to that survey link at any time in the future to make edits to their response. If the Survey Login feature is enabled for the survey, then instead of using a Return Code, they will use their login credentials to return to the survey.
If enabled, participants who have completed the survey will still appear in the Participant List in the Compose Survey Invitations popup to allow them to be invited again to edit their completed response. Additionally, their survey link and survey access code will also remain in the Participant List for this same purpose.
Note: If Survey Notifications have been enabled for a survey that has the "Edit Completed Responses" option enabled, then whenever the respondent returns to the survey again and completes the survey again, it will again trigger the Survey Notifications to send an email to those users selected.
Improvement: Data can now be piped into the URL when using the "Redirect to a URL" option for surveys. This allows users to pass along respondent-specific data in the URL when the respondent gets redirected to a webpage after completing a survey.
Improvement: Data can now be piped into the "href" attribute of an HTML link. In previous versions, if this was attempted, it would not work and would instead create a malformed HTML link that was unusable.
New "Data Exports, Reports, and Stats" module - This single module now replaces the old Report Builder, reports, Data Export Tool, and Graphical Data View & Stats page, so that all their functionality now exist in this new module, including even more new features and functionality.
This new module allows users to create or edit reports, view reports, export all data in the project, export data for a given report, view the descriptive statistics and data plots for a given report, and perform other kinds of exports (e.g., PDF export of all records/all fields, ZIP file export of all uploaded files for all records). Please note that user privileges are applied appropriately throughout this module. For example, if a user has "No Access" data export rights, they will not be given any options to export data, and similarly if they do not have "Add/Edit Reports" privileges, then they will not be given the ability to create, edit, delete, copy, or reorder any reports on the page. Thus the things they are able to do and see in this module depends completely on what their user privileges are.
When performing a data export, users must now choose up front in which format they wish to export their data (e.g. Raw CSV, SPSS, R). This is different from the old Data Export Tool, in which you would choose the format/files *after* the export had completed.
Improvements when creating/editing reports:
Each report now has its own user access setting so that users may choose specific users, User Roles, and/or Data Access Groups that can view that specific report. If a user does not have access to a specific report, then that report will not appear on their left-hand menu in the project. Note: This user access setting does not alter in any way the user's ability to see certain data in the report. It merely gives them access to view the report as a whole, although their user rights might limit the data in the report that they are able to see. For example, if the user is in a DAG, they will only see records assigned to their DAG in the report. And if they do not have form-level access to certain instruments, then they will not be able to view data for fields from those instruments. Thus, reports will not allow users to view data to which they do not already have access.
"Limiters" in reports will now be called "filters" in version 6.0 and higher.
Filters in a report now do not have to be included as a field in the report itself, and they can be set up using complex AND/OR logic. There is even an advanced filter logic format that allows users to hand-type the logic to make it as complex as they wish if the simple format is not powerful enough for what they desire (similar to advanced branching logic syntax).
In the list of projects on the "Data Exports, Reports, and Stats" page, reports can now be easily reordered in a project using drag-n-drop functionality.
Fields included in a report can now be easily reordered within the report using drag-n-drop functionality.
The Data Access Group name and/or the survey identifier and survey timestamp fields can now be displayed in the report, which was not possible in previous versions.
All fields from a whole data collection instrument can be easily added at once to a report by selecting the instrument from a drop-down list. This makes it much faster to add a lot of fields to the report very quickly.
New "Additional Filters" section allows reports to be easily filtered so that they only return records assigned to specific Data Access Groups and/or data from specific events (in longitudinal projects) by allowing the user to select one or more DAGs and/or events from a multi-select drop-down list.
The data displayed in reports can now be ordered by third field, whereas previous versions only allowed sorting by two fields.
Improvement Reports: When building reports in a longitudinal project, Step 3 now contains a new checkbox option: "Show data for all events for each record returned". If this option is CHECKED, then *all* events are returned for each record, but if UNCHECKED, then some events in each record *may* get removed (depending on the filters defined in the report). This option provides users with more control when using report filters, in which it allows them to apply filters to return a group of records (e.g., a cohort) and then optionally filter those records returned even further, such as by removing specific events. This will make report filtering much more palatable for longitudinal projects.
Improvement: When building a report on the "Data Exports, Reports, and Stats" page, a new "Quick Add" button has been added to Step 2, in which it opens a popup window to allow users to select fields for the report very rapidly using the old-style checkboxes (similar to the Data Export Tool in REDCap version 5.X).
Improvement: On the "Data Exports, Reports, and Stats" page, users can now create a new report based on the custom selections made in Report B. This makes it much easier to create a report that includes all fields from many instruments and/or events.
Change: In previous versions, if a user was viewing a report that contained one or more fields that existed on a data collection instrument to which they did not have form-level privileges, then it would simply not show any of the report and display an error message. This was a bit heavy handed. It will now display the report for them, but for any fields that they do not have form-level access to, it will gray those columns out and not display their data. This is much better because it allows the users to view the data to which they do have access.
Change: When exporting the "Return Codes for Surveys" CSV file, which is available on the Additional Export Options tab when the project contains surveys that have the "Save & Return Later" feature enabled, the resulting CSV file no longer includes *all* fields from the project but instead only includes the following fields: Survey Identifier field, Survey Timestamp field for each survey, Survey Return Code field for each survey, Form Status field for each survey, Event Name (if project is longitudinal), and Data Access Group field (if project contains DAGs and the user downloading the file is *not* assigned to a DAG).
Improvement: Double Data Entry users will now be able to fully utilize the new "Data Exports, Reports, and Stats" module, whereas in previous versions, they were not able to access reports or the Data Export Tool. (Note: If data is exported, records will still end with --# in the resulting export file, but reports will display the records without the --# when viewing the report on the webpage in order to maintain consistency in the user interface.)
New user rights privilege for Data Export rights: "Remove all tagged Identifier fields" - If a user is given this data export privilege setting, then it will be applied to *all* data export files in all modules where data is exported (e.g., PDFs, reports, API). In reports and in API data exports, any fields that have been tagged as Identifier fields will simply be removed from the export file. However, in PDF exports, it will include the Identifier fields in the PDF, but it will remove and replace any saved data values with the text [*DATA REMOVED*] for such fields.
New features: New functions exist that can be used in logic for Report filtering, Survey Queue, Data Quality Module, and Automated Survey Invitations. Each of these are explained in detail at the bottom of the Help & FAQ page.
contains() - Does text contain another text string?
not_contain() - Does text not contain another text string?
starts_with() - Does text start with another text string?
ends_with() - Does text end with another text string?
New additions to the API:
See API Help page for full details
"Export PDF file of Data Collection Instruments" - Returns a PDF file of one or all instruments in the project, either with no data (blank), with a single record's data, or with all records from the project.
"Export a Survey Link for a Participant" - Returns a unique survey link (i.e., a URL) in plain text format for a specified record and data collection instrument (and event, if longitudinal) in a project.
"Export a Survey Queue Link for a Participant" - Returns a unique Survey Queue link (i.e., a URL) in plain text format for the specified record in a project that is utilizing the Survey Queue feature.
"Export a Survey Return Code for a Participant" - Returns a unique Return Code in plain text format for a specified record and data collection instrument (and event, if longitudinal) in a project with surveys that are utilizing the "Save & Return Later" feature.
"Export a Survey Participant List" - Returns the list of all participants for a specific survey instrument (and for a specific event, if a longitudinal project).
"Export List of Export Field Names" - Returns a list of the export/import-specific version of field names for all fields (or for one field, if desired) in a project. This is mostly used for checkbox fields because during data exports and data imports, checkbox fields have a different variable name used than the exact one defined for them in the Online Designer and Data Dictionary, in which *each checkbox option* gets represented as its own export field name in the following format: field_name + triple underscore + converted coded value for the choice.
New "Export REDCap Version" API method - Returns the current REDCap version number as plain text (e.g., 4.13.18, 5.12.2, 6.0.0). Set the parameter content="version" in the API request, and it will return the REDCap version number.
New API method for exporting data: "Export Reports" - Any report created by a user can be exported via the API in JSON, XML, or CSV format. Needed for the API request is the report ID number, which is listed next to the report name on the "Data Exports, Reports, and Stats" page (only displayed there if the user has API Export privileges). In order to export a report via the API, the user must have been given access to that report, must not have "No Access" data export rights, and must have API Export rights. More details regarding the API Export Reports method can be found on the API Documentation page.
New optional parameter for API data exports: "rawOrLabelHeaders" parameter for "Export Records" and "Export Reports" API methods. Possible values: "raw" (default) or "label". Parameter is valid for "csv" format only (and only works for "flat" type output). Allows one to have the API return the CSV headers in the data export either as the variable/field names (rawOrLabelHeaders=raw, or if rawOrLabelHeaders is not specified) or instead as the field labels (rawOrLabelHeaders=label).
New API export optional parameter: "exportCheckboxLabels" parameter for "Export Records" and "Export Reports" API methods. Possible values: "true" or "false" (default). Parameter is only valid for "flat" type output. Allows one to set the format of checkbox field values when specifically exporting the data as labels (i.e., when rawOrLabel=true). When exporting labels, by default (without providing the exportCheckboxLabel flag or if exportCheckboxLabel=false), all checkboxes will either have a value "Checked" if they are checked or "Unchecked" if not checked. But if exportCheckboxLabel is set to true, it will instead export the checkbox value as the checkbox option's label (e.g., "Choice 1") if checked or it will be blank/empty (no value) if not checked.
API method "Export Project Information" - Exports some of the basic attributes of a given REDCap project, such as the project's title, if it is longitudinal, if surveys are enabled, the time the project was created and moved to production, etc. See the official API documention/help page in 6.5.0 for all the details.
Survey Invitation Log
Improvement: The Survey Invitation Log has a different default display, in which it now defaults to displaying all future invitations when the page is first loaded and displays all invitations by send time in ascending order. In previous versions, it displayed them in descending order by default, and also displayed all invitations prior to 11:59pm of the current day. This should make it less confusing for users when they first load the page.
Improvement: The Survey Invitation Log now has two buttons ("View past invitations" and "View future invitations") at the top of the table on the page to allow users to quickly view past or future invitations without having to use the "begin time" or "end time" filter, which can be a bit more complicated.
New feature: Users may now download CATs (computer adaptive tests) from the REDCap Shared Library. Over three dozen of these CATs have been added to the Shared Library, in which these include PROMIS instruments provided by the Assessment Center API software ( https://www.assessmentcenter.net/). Once downloaded from the Shared Library, CATs are rendered in REDCap as surveys that are displayed one question at a time. CATs are dynamic and adaptive, which means that the next question to be displayed in the CAT survey is decided upon by the previous answers in that survey, in which a third-party web service (hosted by a server at Vanderbilt University in a way similar to how the REDCap Shared Library is hosted) is utilized to provide this dynamic functionality seamlessly for the survey participant. For more information on PROMIS instruments, see http://www.nihpromis.org/.
Improvement: The choice options for radio button fields and drop-down fields are now properly displayed as circles in PDF exports rather than as squares, which would often make it difficult to discern if a field was a checkbox or radio/drop-down when viewing the PDF.
Improvement: Double Data Entry now works with the Scheduling module and also works with surveys that are initiated by clicking the link on a data entry form.
Data Resolution Workflow / Field Comments
Improvement: Field Comments may now be edited and deleted. For all existing projects and all new projects created, the ability to edit/delete a field comment will be enabled by default. If users do *not* wish to allow this functionality, they may disable it for the project in the Optional Customizations popup on the Project Setup page.
Improvement: For projects using the Data Resolution Workflow (data queries module), two new user privileges were added: 1) Open queries only, and 2) Open and respond to queries. In previous versions, if a person were given the ability to open queries, it also included their ability to close and respond to queries. These new user privileges allow users to just open or both open and respond to queries without having the ability to close them.
BUGS & OTHER CHANGES:
Major bug fix: When utilizing datediff() and some other advanced logic functions in the logic used in Data Quality rules, Automated Survey Invitations, Survey Queue, report filters, etc., it might mistakenly evaluate the logic as true when it should return false. This could cause Automated Survey Invitations to get sent prematurely or cause surveys to appear in the Survey Queue prematurely, among other things.
Improvement: When viewing reports in longitudinal projects, any fields displayed in the report that are not designated for that particular event (i.e., row in the report) will be grayed out to show that the field is not designated. This makes it easier for users to discern if a field's value is not applicable or if it is missing.
Change: Record auto-numbering is enabled by default for new projects that are created from scratch.
Change: When creating Project Bookmarks in a project (or Custom Application Links in the Control Center), the "Link URL / Destination" is no longer required to be full URL (beginning with "http") but can now be a relative link (i.e., beginning with "/") or can begin with another non-http protocol. This adds flexibility for users creating bookmarks.
Bug fix: When using the Data Comparison Tool for a project with Double Data Entry enabled, if the records entered by DDE person 1 and person 2 had a record name that was in a different case than the other in the pair, then it would mistakenly not allow the user to compare the pair of records nor merge them together.
Bug fix: When using the Scheduling module in a longitudinal project containing more than one arm, in which either the Custom Record Label or Secondary Unique Field is being used, then it would mistakenly display the Custom Record Label or Secondary Unique Field for only the records from the first arm. For all other arms, there would not be a value displayed for the Custom Record Label or Secondary Unique Field for a given record in that arm.
Change/bug fix: When editing the record ID field (i.e., first field in project) in the Online Designer when record auto-numbering has been enabled in the project, it would not allow you to set the validation as "Integer" even though "Integer" is technically a valid option when record auto-numbering is enabled. It will now allow you to set it as "Integer" but will display an error message if changed to any other validation type. In previous versions, the field validation for the record ID field could be changed via the data dictionary but not in the Online Designer.
Bug fix: For longitudinal projects that utilize surveys in which a survey is not designated for the first event in the project, when a user first navigates to the Participant List page (i.e., by clicking the "Participant List" tab), it would mistakenly not have the first option selected in the drop-down list of surveys in the Participant List. This might cause the user to think they are viewing the Participant List of a different survey if they are not paying close attention.
Improvement: It now does a better job of not removing "<>", "<=", or "<"+number in text, which in previous versions might mistakenly get removed on certain pages (e.g. field labels on forms/surveys) because they were assumed to be an illegal HTML tag.
Bug fix: On certain rare occasions, such as when a user is assigned to a Data Access Group and records have been created via a data import of only the record ID field and no other fields, the Record Status Dashboard might mistakenly not display all the records that it should on the page.
Bug fix: Any users that were suspended from the system but still had a user-level expiration date set would mistakenly receive an email warning letting them know that their account would expire soon, which is confusing because their account had already been suspended. It now no longer sends the expiration warning email to suspended users.
Change: The font size of some of the text on the main login form was increased.
Bug fix: When adding new arms or renaming arms on the Define My Events page in a longitudinal project, certain non-Latin (UTF-8) characters would not save correctly in the arm name, thus resulting in a black question mark symbol for that character.
Bug fix: When viewing a PDF export of a survey or data entry form in which non-Latin (UTF-8) characters are used in a Slider field's field label or in its slider labels, the text would not display correctly in the PDF.
Bug fix: When creating a new instrument from scratch in the Online Designer, if a new instrument's name is very close to an existing instrument's name, in which the name is also very long (>50 characters), then there is a chance that when the instrument is being created it will get stuck in an infinite loop and never actually create the instrument.
Change: The "Help & FAQ" page was updated.
Change/improvement: When using the Scheduling module in a project with multiple arms, if a user chooses to schedule an existing record by selecting the record from the drop-down on that page, it will now auto-select the arm in the arm drop-down based on the arm on which that record exists. If the record exists on multiple arms, it will auto-select the first arm on which it exists. This will prevent users from inadvertently scheduling a record on the wrong arm while also giving the user the flexibility to schedule the record on another arm, if they choose to do so, why changing the arm selection from the one that is auto-selected for them.
Change: When the recipient of a Send-It file loads the webpage to download their file, the password field now has the autocomplete="off" attribute to prevent web browsers from possibly auto-filling the password, if the browser had somehow stored it.
Bug fix: When using record auto-numbering in a longitudinal project, there is a chance that when two users are creating a record at the same time in which one user clicks the "Save and Continue" button on the form to create the record, they will mistakenly get locked out of that record until the other user saves their record and leaves the data entry form. It does not appear that data gets overwritten for the record nor that records get duplicated. It merely locks a user out of a record mistakenly when it should not.
Bug fix: The field validation type "Phone (U.S.)" was modified to be more inclusive of more area codes. Also, it was renamed to "Phone (North America)" since it actually applies to all of North America and not just the United States.
Change/improvement: When navigating to the Data Quality module in a project, the record drop-down list on that page now loads via AJAX after the page has been fully loaded. This makes the page load much faster for projects containing lots of records.
Change/improvement: The Stata syntax file that is produced during data exports now explicitly defines the minimum version as Stata 12. This reduces the number of issues that might occur when loading REDCap data into Stata by setting the first line of the syntax file as "version 12".
Bug fix: When a user that is assigned to a Data Access Group is accessing a longitudinal project via the mobile web view, it would mistakenly display all the records for the project in the record drop-down list on the page when instead it should only display the records belonging to the user's DAG. If the user clicked any records not in their DAG, it would not allow them to access them, as expected.
Change: When uploading a data dictionary containing a Text field with a min or max validation range in which the validation type of the field is a "number_Xdp" validation, it will accept values for the min or max if it is a number even though it does not follow the validation rule explicitly (e.g., it will allow a min of 5.6 even though the validation requires 3 decimal places). This is done because Microsoft Excel often removes any trailing zeros after decimals when opening the data dictionary and re-saving it. So this serves as a workaround for Excel's undesirable behavior without compromising anything with regard to data quality.
Bug fix: The "email" field validation type was modified to be more inclusive so that it now allows for plus signs (+) in email addresses, which is allowable for certain email services.
Bug fix: Some language was mistakenly not abstracted (i.e., was hard-coded in English) in the "Return Code" popup on a survey's "Save & Return Later" page.
Bug fix: The JavaScript functions that enable Slider fields and reset their value were mistakenly employing branching logic prior to calculations when it should have been performing calculations first for calc fields.
Change: When the API crashes due to exceeding web server memory during a data export or import, it now returns a 500 error code with the message "REDCap ran out of server memory. The request cannot be processed. Please try importing/exporting a smaller amount of data." In previous versions, it would return different status codes, which could be confusing and not helpful for the user to troubleshoot the situation.
Change: The file_import.php file inside the API Example ZIP file was updated to work with PHP 5.5.0 and higher.
Bug fix: When a Table-based user resets their own password and is setting a new one, in which they attempt to enter a password that has fewer than 9 characters, which is the minimum, it displays an error message that mistakenly says it must be "10 characters" at minimum when it should instead say "9 characters".
Bug fix: If the Participant List for a survey contains more than 50 participants, and a user goes to remove a participant, then after removing the participant, it would mistakenly reset the list back to the first page of participants rather than keeping it on the current page of participants. This would make it very difficult to remove many participants in some cases.
Change: The API method "Export Users" will now return two new attributes for each user: "mobile_app" and "mobile_app_download_data". If mobile_app's value is "1", then the user has privileges to use the REDCap Mobile App for that project. If "0", then not. If mobile_app_download_data's value is "1", then the user has the ability to download all records from the project to the mobile app, but if "0", then the user will not have the option in the app to download any records to the app.
Improvement/bug fix: If a survey's title is blank (has no text), then the row for this survey in the Survey Queue would show up as blank, which could be confusing to participants. This also occurs when in the Automated Survey Invitation setup popup, in which the survey's option shows up as blank in the drop-down list of surveys. In these cases, it now displays the instrument name in place of the title if the title is blank.
Change: Replaced the "Detailed Overview" video
Bug fix: When importing a file via the API File Import method for a longitudinal project in which the event to which the file is being imported does not yet have any data, although the record does exist, it would mistakenly give the error message "The record 'XX' does not exist. It must exist to upload a file", which is not true. This error would prevent the file from being uploaded.
Major bug fix: If a survey respondent completes the last page of a multi-page public survey and then they click the Back button in their browser and then reload the survey page, it would mistakenly allow them to re-submit their survey results and even change some survey responses after having completed the survey already. This does not apply to one-page surveys and does not apply to unique survey links.
Bug fix: In a longitudinal project, if Automated Survey Invitations are set up for surveys not on the first event, and then the project is converted back to a non-longitudinal project, it would mistakenly schedule and send invitations for those events that are now orphaned and no longer utilized in the classic project.
Bug fix: If a project has surveys enabled and has an active survey in which participants have already received a survey link to take the survey, then if a user in the project disables surveys in the project via the "Main project settings" section on the Project Setup page, the surveys will mistakenly still load and function normally for participants when instead they should display an error message that the survey is not active.
Change: The "Export PDF" API method's optional parameter "allrecords" has been changed to "allRecords" to be more consistent with API parameter naming conventions. Note: To be backward compatible, the older version "allrecords" will still work the same as before if it is used.
Change: If a project contained only one data collection instrument, then a user would mistakenly be able to delete the instrument on the Online Designer. It now prevents the user from deleting the only instrument but instead recommends that they add a new instrument and then delete the old one.
Bug fix: When sending a survey invitation that utilizes piping in the content of the email (when sending the invitation from the Participant List, Automated Survey Invitation, or top of data entry form), then if the user later went to the Participant List to compose another invitation and clicked the "load message box with text from a previous email?" option, it would mistakenly include some HTML "span" tags around the piped data value when loading in the message from that previous email.