Csvbulkload import array into dataobject

Questions
1

Silverstripe Version: 3.6.6

As external data is stored like LinkName (LinkUrl), LInkName (LinkUrl), LInkName (LinkUrl), etc. in one field there is no easy column mapping possible. How do I get that data into a DataObject using the CsvBullkLoader?

I need to import link name and link from an external database. The database field there stores the data like LinkName (LinkUrl), LInkName (LinkUrl), LInkName (LinkUrl) etc. in one field. There can be none or many LinkName(Links) in that field. I only need to store the LinkName and LinkUrl in Silverstripe. In the template I will use these information and the data shows via this link. So no upload from actual files needed.

What I have done so far is:
I created a ProductImage DataObject with Title for LinkName and LinkUrl for LinkUrl. This has a $has_one relationship with a Product DataObject.

I created the ProductImageAdmin ModelAdmin and ProductImageCsvBulkLoader CsvBulkLoader extensions. With the CsvBulkLoader extension I have the problem not knowing how to map the fields as I don’t have a one to one mapping.
I know I need to somehow involve an array in this, but I don’t know how.

Here is my code so far which would work if I had a one to one mapping:

public $columnMap = array(
        'Extra Pictures' =>'->importImageInfos',
        'ProductFamily' => 'Product.Title'
    );
public static function importImageLink(&$obj, $val, $record){
        $imageArray = explode(',', $val);
foreach($imageArray as $subval){
            $parts = explode('(', $subval);
            
            $imgName = $parts[0];
            $newpart = explode(')', $parts[1]);
            $imgUrl = $newpart[0];

            $obj->Title = $imgName;
            $obj->ImageLink = $imgUrl;
}
public $relationsCallbacks = array(

        'Product.Title' => array(
            'relationname' => 'Product',
            'callback' => 'getProductByTitle'
        )
    );

    public static function getProductByTitle (&$obj, $val, $record){
        return Product::get()->filter('Title', $val)->First();
    }

public $duplicateChecks = array(
		'Extra Pictures' => 'ImageLink'
    );

column name: Extra Pictures (luckily it is a column in the middle of the file)
data example in colum field: ER-1.png (https://domain.com/3FSduKOJzR_ER-1.png),ER-2.png (https://domain.com/TzGbAAFNlSxC_ER-2.png),ER-4.jpg (https://domain.com/L9JLcLfRWG_ER-4.jpg)

It is not a one-off import. I need to be able to do it whenever I need to update the data.

This works fine to get one of the images as it gets overwritten, so I need to add somehow an array, but I don’t know how to create this in order to fill my DataObject from the CsvBulkLoader extension.
The more I read about ArrayList, ArrayData and DataObject google it the more I get confused.

I have to say that I am relatively new to PHP and Silverstripe. The solution might be easy but I am so confused that I just don’t get this.

Any help would be much appreciated.

Thank you very much.

2

Will you be doing the import once or will it be a repeatable action?

Also could you put down a data row example? I am seeing brackets being used somewhere as a delimiter?

3

The import needs to be a repeatable action. I added additional information in my post regarding data information.
Thank you

4

This is what I have working in a SS4, you can probably take reference from here:

/* TestProduct.php */
namespace Test\Models;

use SilverStripe\ORM\DataObject;

class TestProduct extends DataObject {

    private static $table_name = 'Test_TestProduct';

    private static $db = [
        'Title' => 'Varchar(255)',
    ];

    private static $has_many = [
        'Images' => TestProductImage::class,
    ];

    private static $summary_fields = [
        'Title' => 'Title',
    ];

}

/* TestProductImage.php */
namespace Test\Models;

use SilverStripe\ORM\DataObject;

class TestProductImage extends DataObject {

    private static $table_name = 'Test_TestProductImage';

    private static $db = [
        'Title' => 'Varchar(255)',
        'URL' => 'Varchar(255)',
    ];

    private static $has_one = [
        'Product' => TestProduct::class,
    ];

    private static $summary_fields = [
        'Title' => 'Title',
        'URL' => 'URL',
    ];

}

/* TestProductAdmin.php */
namespace Test\Extensions;

use Test\Models\TestProduct;
use Test\Models\TestProductImage;
use SilverStripe\Admin\ModelAdmin;

class TestProductAdmin extends ModelAdmin {
    
    private static $menu_title = 'Tests';
    private static $url_segment = 'tests';
    private static $menu_icon_class = 'font-icon-happy';
    private static $menu_priority = 4;

    private static $managed_models = [
        TestProduct::class,
        TestProductImage::class,
    ];

    private static $model_importers = [
        TestProduct::class => TestCsvBulkLoader::class,
    ];
}

/* TestCsvBulkLoader.php */
namespace Test\Extensions;

use Test\Models\TestProduct;
use Test\Models\TestProductImage;
use SilverStripe\Dev\CsvBulkLoader;

class TestCsvBulkLoader extends CsvBulkLoader
{
    /* Note: expects import columns: ProductName, ProductImageInfo
     * e.g.:
     * MyProduct, "Name1 (Link1), Name2 (Link2), Name3 (Link3)"
     */
    public $columnMap = [
        'ProductName' => 'Title', 
        'ProductImageInfo' => '->importImages',
    ];
    public static function importImages(&$obj, $val, $record) 
    {
        $parts = explode(',', $val);
        $obj->write(); // write first in order to generate its ID for use in the image relation
        
        foreach($parts as $part) {
            $part = str_replace(['(', ')'], ',', $part);
            $subparts = explode(',', $part);
            if(count($subparts) < 2) {
                continue;
            }
            $image = new TestProductImage();
            $image->Title = trim($subparts[0]);
            $image->URL = trim($subparts[1]);
            $image->ProductID = $obj->ID;
            $image->write();
        }
    }
}

The main trick is in importImages(), where we have to call $obj->write() once first, since at the point in time the main Product object has not yet been created, you wouldn’t be able to link the relation between it and the images. Writing thus allows its ID to be used to define that relation.

5

Wow, thanks CW_Chong!
This first $obj->write(); is the important thing!
How would you go about a many-many relationship with CsvBulkLoad? How do you get those connected as you have to start with one csv without having the data of the other to connect. Do you enter the ID of the first into the ProductCategoryProduct table and then import the other csv and get the ID from that one and look for matching ID and write it into the ProductCategoryProduct table where there already is the matching ID?
Thank you

6

Should be similar to the example

Imstead of looping the images and creating them outright, you would

  1. Loop through categories
  2. Find existing category by name; if fail, create and write a new one;
  3. Push id of category from 2. Into the products’ category list; write (this should create the correct entry in join table if the relationship is defined correctly)
7

Hi CW_Chong,
Thanks for your immediate reply.
They changed the relationship from ProductCategory to Products to Products to ModelInfos as a many_many relationship.
The table product_modelinfos hasn’t filled itself by this, but after a bit more googling I found this

     $obj->Products()->add($product);

So, everything looks like:
$obj->write(); // write first in order to generate its ID for use in the relation

    if (!empty($parts)){
        foreach($parts as $part) {
            
            $product = Product::get()->filter('Title', $parts)->First();
            
        if (!empty($product)){
                $product->ModelInfoID = $obj->ID;
                $product->write();

                $obj->Products()->add($product);
         }

and now it all works fine.

I think, I was thinking too complicated initially.
So, thank you very much for pointing me in the right direction and all your help!!