Potential Missing data from form submissions in Oct 9-14 2020

On Friday, October 9th, we deployed some code that accidentally prevented updates to forms relying on our form handling service. Below is an outline of what happened, and what we’ve done to fix the problem, and what work remains.

If you had a previously working form that was deployed before Friday 9 Oct @ 20:28 UTC, and which you changed by adding, removing, or renaming one more more input fields, between then and now (Wednesday 14 Oct at 17:42 UTC), it is likely we did not correctly process your update, and have not been collecting the visitor-submitted data from those fields since last Friday.

We know that you rely on forms and form inputs, and we’ve been working hard to fix the issue. We have retroactively processed form updates to production deploys on all affected sites, so all of those forms have already begun receiving data for the changed-field submissions.

Please note that we have not re-processed all deploys during that time period, only the current production deploy. However, any other deploy previews or branch deploys with form changes during that time would also have been affected, and can be repaired by you creating a new deploy on that branch or PR (e.g. re-deploying staging-form on your staging branch that is NOT part of the production deploy). Both new commits or new CLI deploys will trigger the repair-on-redeploy behavior. If you have any questions about how to accomplish this, please ask below and we will advise.

Data from submissions that your visitors have attempted to submit which was in these “missing” (changed) fields was not saved during this time window, and thus cannot be restored. Data in any fields present before Friday was collected and notified about as usual, if submitted.

We want to acknowledge the impact this may have - we never want our customers to have the experience of potentially losing data. We are completing a thorough analysis of what went wrong here, and are taking steps to prevent this from re-occurring.

If you have any questions, concerns, or need clarification on anything, please comment below and we’ll respond as soon as we can.

6 posts were split to a new topic: Forms not picking up new fields

I have a site with a form that I’m pretty sure was effected by this, but possibly in a different way. Here’s what I got.

  • A standard email sign up form using Netlify Forms
  • Originally deployed sometime in Sept 2020, so before Friday 9 Oct @ 20:28 UTC
  • Collected an email field only
  • Data being collected as expected
  • On Oct 13, 2020, 5:54 PM EDT we deployed a change to the form that added a tags field.
  • On Oct 20, 2020 at about 6:30 PM EDT we saw what looked like a gap in Verified submissions data on the form dashboard.
  • We had verified submission at Oct 13 at 4:31 PM then a gap until Oct 19 at 1:27 PM

We also had an email notification set up for the form. With that, I was able to look back at the emails and see that we had signups in that gap. Luckily this is a low-volume form. I was able to go through manually and find 15 missing sign ups.

Leaving out the email address out for privacy, but here are the times they came through:

  • Tue, Oct 13, 7:05 PM
  • Wed, Oct 14, 3:21 AM
  • Wed, Oct 14, 7:37 PM
  • Wed, Oct 14, 9:23 PM
  • Thu, Oct 15, 8:44 AM
  • Thu, Oct 15, 9:03 AM
  • Thu, Oct 15, 11:57 AM
  • Thu, Oct 15, 1:57 PM
  • Thu, Oct 15, 4:04 PM
  • Thu, Oct 15, 4:22 PM
  • Thu, Oct 15, 5:18 PM
  • Fri, Oct 16, 12:15 PM
  • Fri, Oct 16, 1:12 PM
  • Fri, Oct 16, 7:21 PM

Then the next one was Oct 19 at 1:27 PM. So it looks like our form may have been fully down for the 17th, 18th and maybe part of the 19th, like you said in your post because we didn’t get any email notifications for those dates.

Again, we got the email notification for each of those form submission. And I don’t have a screenshot so I can’t say for certain, but I’m pretty sure I had seen some. or all of those submissions in the Verify submissions list of the form dashboard.

It seems like that data made it in via a form submission, but then was lost. I’m wondering if that is also a side-effect of the issues. Does that sound right @fool

Hi Tyler,

Could you let us know the URL to which your form is submitted in production? Or if you prefer not to share that in public, at least the site’s API ID (from the general settings tab) and the form ID (from the end of the URL for looking at its submissions in our UI). Those are safe to share publicly/don’t leak any sensitive information.

With that detail, we can examine our internal logs to see if we can understand what happened.

Thanks in advance for your

Thanks, sure thing:

Can share any other details as needed.

Thanks! I’ll ask our team to look into this for us.

Hey @tyler.gaw,

Our backend team looked in to this for you. It transpires that 69 records were deleted. However, these were deleted by members within the team, from the UI.

21 records were removed by an email address starting tyler[...], 48 from me[...]. I hope this helps!

Thank you. Yeah we have deleted records from the UI. Those that you mentioned I know were test submissions. Those are known. But those do not include the detailed list I described above. The ones above, to the best of my knowledge, were not deleted by myself or the one other team member with access to do so through through the UI.

Hey @tyler.gaw,

Our backend team have looked in to the particular timeframe given and observed that the records in the list provided were deleted.

For example, the Tue, Oct 13, 7:05 PM EDT (2020-10-13 23:05 UTC) record was deleted by me[...] at 2020-10-20 16:03 UTC.

All records in this list appear to have been deleted by the me[...] user. Hope this helps!

Hi there

Got sent here from my original thread. I just wanted to know if any progress was made on this.

Cheers,
Pierre

Hi Pierre,

Not sure what progress you are expecting? As far as I can tell from this thread:

  • we established that all lost records matched our time period
  • and that no other records were lost, though some were deleted by customers.

We don’t have any way to get any of the lost data back; it is truly gone. But also all sites should have been fixed so that new submissions have new fields since mid-october. Let me know more about what is still missing from new posts, if anything, please!