Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug(dexie-export-import): importDb data corrupt on non-ASCII string #2105

Open
cha0sCat opened this issue Dec 3, 2024 · 0 comments
Open

Comments

@cha0sCat
Copy link

cha0sCat commented Dec 3, 2024

exportDB works great.

importDB can cause data corruption on non-ascii characters.

For example, if we save '让我们说中文' as string, export & import can make some of the records become '让我们��中文'

Here's the reproducible code:
https://jsfiddle.net/j58xz679/10/

const testDB = new Dexie('testDB')

testDB.version(1).stores({
  testTable: '++id'
})

// generate some test record and insert it
const insertTestData = async () => {
  console.log('start insert test data')

  // const testPattern = 'Lets speak English\n'  // English
  // const testPattern = '日本語を話しましょう\n'  // Japanese
  const testPattern = '让我们说中文\n'  // Chinese
  const data = Array(1000).fill(0).map(() => ({
    content: testPattern.repeat(1000)
  }))
  await testDB.table('testTable').bulkAdd(data)

  console.log('insert test data done', data.length, await testDB.table('testTable').count())
}

// check if there's �� in content (this should not happen)
const checkTestData = async () => {
  console.log('start check test data')

  const data = await testDB
      .table('testTable')
      .toArray()
  const corruptedData = data.filter(record => record.content.includes("��"))
  if (corruptedData.length > 0) {
    console.error('check test data failed, corrupt count: ', corruptedData.length)
    return false
  }

  console.log('check test data done, no corrupt', data.length)
}

const exportAndImportDb = async () => {
  // export and import the db
  // export works OK
  console.log('start export db')
  const data = await exportDB(testDB, {prettyJson: false})
  console.log('export db done')

  // proof that export is ok
  const stringData = await data.text()
  if (stringData.includes('��')) { console.warn('exported data should not contain ��') }

  // !!! this is where the bug happens
  console.log('start import db')
  await importDB(data, {clearTablesBeforeImport: true})
  console.log('import db done')
}

testDB.table('testTable').clear()
    .then(() => insertTestData())
    .then(() => checkTestData())  // this should success (no corrupt)
    .then(() => exportAndImportDb())  // this should cause the data to be corrupt
    .then(() => checkTestData())  // this should fail (corrupt data)
@cha0sCat cha0sCat changed the title Bug(dexie-export-import): importDb can cause data corrupt Bug(dexie-export-import): importDb data corrupt on non-ASCII string Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant