ImportExport: Running multiple data imports in a single script can cause errors

[piotrekkaminski]

From @koenner01 on November 3, 2017 9:34

When running multiple data imports in a single script (we are first running a category import, then several product imports), problems can occur.

We are using a script that uses a \Magento\ImportExport\Model\ImportFactory to create \Magento\ImportExport\Model\Import objects for each individual import. But because the Import model object is using a singleton implementation of \Magento\ImportExport\Model\ResourceModel\Import\Data, the data is shared between different imports. This causes problems for the second import because the previous data isn't cleared.

Preconditions

1. PHP7.0
2. M2.1.x and M2.2.x (CE)

Steps to reproduce

1. Create a script that uses at least 2 imports (we are running a loop that creates Import model objects)
   example (pseudo code-ish)

public function __construct(
    \Magento\ImportExport\Model\ImportFactory $importModelFactory,
    ...
)
{
    $this->importModelFactory = $importModelFactory;
    ...
}

public function createImportModel()
{
    $importModel = $this->importModelFactory->create();
    $importModel->setData($this->settings); // array containing entity type, behavior, etc.
    return $importModel;
}

public function runImports()
{
    foreach($importData as $data) {
        $importModel = $this->createImportModel();
        ...
        $importModel->importSource();
    }
}

2. Run script with some basic data (for testing purpose we are running 2 arrays of products, containing data for: sku, product_type, attribute_set_code, store_view_code, product_websites, qty, is_in_stock. The products should be non-existent in the webshop as problems only occur when creating new products.

Expected result

1. Every imports smoothly

Actual result

1. Errors are being thrown; For example, during our second import (which is a simple product import) we are getting sql insert errors

1048 Column 'product_id' cannot be null, query was: INSERT INTO `cataloginventory_stock_item` (`manage_stock`,`use_config_manage_stock`,`qty`,`min_qty`,`use_config_min_qty`,`min_sale_qty`,`use_config_min_sale_qty`,`max_sale_qty`,`use_config_max_sale_qty`,`is_qty_decimal`,`backorders`,`use_config_backorders`,`notify_stock_qty`,`use_config_notify_stock_qty`,`enable_qty_increments`,`use_config_enable_qty_inc`,`qty_increments`,`use_config_qty_increments`,`is_in_stock`,`low_stock_date`,`stock_status_changed_auto`,`is_decimal_divided`,`product_id`,`website_id`,`stock_id`) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE `manage_stock` = VALUES(`manage_stock`), `use_config_manage_stock` = VALUES(`use_config_manage_stock`), `qty` = VALUES(`qty`), `min_qty` = VALUES(`min_qty`), `use_config_min_qty` = VALUES(`use_config_min_qty`), `min_sale_qty` = VALUES(`min_sale_qty`), `use_config_min_sale_qty` = VALUES(`use_config_min_sale_qty`), `max_sale_qty` = VALUES(`max_sale_qty`), `use_config_max_sale_qty` = VALUES(`use_config_max_sale_qty`), `is_qty_decimal` = VALUES(`is_qty_decimal`), `backorders` = VALUES(`backorders`), `use_config_backorders` = VALUES(`use_config_backorders`), `notify_stock_qty` = VALUES(`notify_stock_qty`), `use_config_notify_stock_qty` = VALUES(`use_config_notify_stock_qty`), `enable_qty_increments` = VALUES(`enable_qty_increments`), `use_config_enable_qty_inc` = VALUES(`use_config_enable_qty_inc`), `qty_increments` = VALUES(`qty_increments`), `use_config_qty_increments` = VALUES(`use_config_qty_increments`), `is_in_stock` = VALUES(`is_in_stock`), `low_stock_date` = VALUES(`low_stock_date`), `stock_status_changed_auto` = VALUES(`stock_status_changed_auto`), `is_decimal_divided` = VALUES(`is_decimal_divided`), `product_id` = VALUES(`product_id`), `website_id` = VALUES(`website_id`), `stock_id` = VALUES(`stock_id`)

Notes:

We have found a fix for our situation by doing:

public function createImportModel()
{
    $importModel = $this->importModelFactory->create();
    $importModel->setData($this->settings); // array containing entity type, behavior, etc.
    $importModel->getDataSourceModel()->getNextBunch();
    return $importModel;
}

By adding the getNextBunch function, the iterator on the data is being reset between each import. This is far from ideal but it works.

The best fix would be to add a \Magento\ImportExport\Model\ResourceModel\Import\DataFactory implementation to \Magento\ImportExport\Model\Import instead of using the singleton approach but this requires an overhaul on the core files..

This issue is also not always reproducible, which will probably be the reason why this issue will be closed; But hopefully someone who stumbles on the same problem as us finds any use in our fix.

Copied from original issue: magento/magento2#11990

[piotrekkaminski]

From @koenner01 on November 3, 2017 12:59

Ok my previously suggested fix doesn't work; I'm running into the same kind of problems when importing 10k+ products.

I'm now using the following in my module di.xml

<type name="Magento\ImportExport\Model\ResourceModel\Import\Data" shared="false" />

This forces every instance of the object to not be shared in the objectManager (and thus making it 'non singleton')