You are here:  » Bulk add and register feeds

Active Forum Topics


Bulk add and register feeds

Submitted by macland on Mon, 2016-10-24 19:14 in

I have a list of 161 feed urls, and rather than getting a VA to add all those to the automation tool and then register them I was wondering if there was a script to automate this?

The feeds are all set up the same way, and all fields are being auto detected during import so theres no manual judgement or anything needed

Submitted by support on Tue, 2016-10-25 09:34

Hi,

Does your list include any meta data, specifically, merchant name?

Or would you be able to include that easily, for example by including the merchant name comma separated from the URL e.g.

Merchant 1,http://www.example.com/m/1111/feed.asp
Merchant 2,http://www.example.com/m/2222/feed.asp
Merchant 3,http://www.example.com/m/3333/feed.asp

If that's the case, or you can post an example (with example URLs in place of the actual) it would be no problem to scan the file and automatically create pt_feeds entries based on a supplied template (already registered) feed...

Cheers,
David.
--
PriceTapestry.com

Submitted by macland on Thu, 2016-10-27 09:02

Hi thank you for your reply.
I could add it easily. but it's also found inside the feed.

Not sure if I correctly understand what you mean with example, but the feeds are just normal feed urls like this

{code saved}

Submitted by support on Thu, 2016-10-27 11:20

Hi,

I'll look at including a generic "bulk register" utility in a later distribution, in the mean time, as the merchant name is in the feed, if you are happy for filenames to be set-up as

parnter-ads106.xml
parnter-ads116.xml
parnter-ads119.xml

etc.

...have a go with the following, saved as admin/bulk.php:

<?php
  set_time_limit
(0);
  require(
"../includes/common.php");
  require(
"../includes/admin.php");
  require(
"../includes/automation.php");
  require(
"../includes/MagicParser.php");
  require(
"admin_header.php");
  if (isset(
$_POST["feeds"]))
  {
    
$now time();
    
$feeds explode("\n",trim($_POST["feeds"]));
    foreach(
$feeds as $feed)
    {
      
$feed trim($feed);
      
$parts parse_url($feed);
      
parse_str($parts["query"],$vars);
      
$filename "partner-ads".$vars["feedid"].".xml";
      print 
$filename."<br />";
      
$sql "DELETE FROM `".$config_databaseTablePrefix."jobs` WHERE filename='".database_safe($filename)."'";
      
database_queryModify($sql,$result);
      
$sql "INSERT INTO `".$config_databaseTablePrefix."jobs` SET
                directory = 'feed',
                filename = '"
.database_safe($filename)."',
                url = '"
.database_safe($feed)."',
                unzip = '0',
                minsize = ''
                "
;
      
database_queryModify($sql,$id);
      
automation_run($id);
      
$sql "DELETE FROM `".$config_databaseTablePrefix."feeds` WHERE filename='".database_safe($filename)."'";
      
database_queryModify($sql,$result);
      
$sql "INSERT INTO `pt_feeds` SET
                filename = '"
.database_safe($filename)."',
                registered='"
.$now."',
                format='xml|PRODUKTER/PRODUKT/',
                field_merchant='FORHANDLER',
                field_name='PRODUKTNAVN',
                field_description='',
                field_image_url='BILLEDURL',
                field_buy_url='VAREURL',
                field_price='NYPRIS',
                field_category='KATEGORINAVN',
                field_brand='BRAND'
                "
;  
      
database_queryModify($sql,$result);
      
admin_import($filename,10);
    }
    print 
"Done.";
    exit();
  }
  print 
"<form method='POST'>";
  print 
"<textarea name='feeds' style='width:100%;height:300px;'></textarea>";
  print 
"<input type='submit' />";
  print 
"</form>";
  require(
"admin_footer.php");
?>

Dry run first with just 2 or 3 feeds and double check that they have been registered and imported as expected (10 product trial import) before running with the full list).

Be aware that the network server limits feed retrieval to one request per day per feed, so make sure that you remove the test feeds before running with the full list - note that I tested this with ID's 106/116/119 from your previous reply...

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by macland on Mon, 2016-10-31 12:19

Thank you very much it works like a charm :-)

Submitted by macland on Thu, 2020-04-30 07:54

Now I actually do need the feeds to be imported as merchantname.xml instead of the partner-ads url.

Would that be easy enough to change?

So I would input like this

{code saved}

and then saved as {code saved}?

If it can only handle a-z the rest can just be stripped, just need something to identify the feed of the merchant, so {code saved} will also be fine :-)

Thank you very much in advance

Submitted by support on Thu, 2020-04-30 09:01

Hi,

This can be done with only a couple of changes to the above, but make sure to use the complete feed URL as the URL part of each line e.g.

Merchant 1,https://www.example.com/path/to/feed.asp?id=1234
Merchant 2,https://www.example.com/path/to/feed.asp?id=5678

The following will then use the tapestry_normalise() function to make the merchant name safe, and swap spaces with underscore...

<?php
  set_time_limit
(0);
  
setlocale(LC_ALL,"en_US.utf8");
  require(
"../includes/common.php");
  require(
"../includes/admin.php");
  require(
"../includes/automation.php");
  require(
"../includes/MagicParser.php");
  require(
"admin_header.php");
  if (isset(
$_POST["feeds"]))
  {
    
$now time();
    
$feeds explode("\n",trim($_POST["feeds"]));
    foreach(
$feeds as $feed)
    {
      
$feed trim($feed);
      if (!
$feed) continue;
      
$parts explode(",",$feed);
      
$filename str_replace(" ","_",tapestry_normalise($parts[0])).".xml";
      
$filename iconv("UTF-8","ASCII//TRANSLIT",$filename);
      
$url $parts[1];
      print 
$filename."<br />";
      
$sql "DELETE FROM `".$config_databaseTablePrefix."jobs` WHERE filename='".database_safe($filename)."'";
      
database_queryModify($sql,$result);
      
$sql "INSERT INTO `".$config_databaseTablePrefix."jobs` SET
                directory = 'feed',
                filename = '"
.database_safe($filename)."',
                url = '"
.database_safe($url)."',
                unzip = '0',
                minsize = ''
                "
;
      
database_queryModify($sql,$id);
      
automation_run($id);
      
$sql "DELETE FROM `".$config_databaseTablePrefix."feeds` WHERE filename='".database_safe($filename)."'";
      
database_queryModify($sql,$result);
      
$sql "INSERT INTO `pt_feeds` SET
                filename = '"
.database_safe($filename)."',
                registered='"
.$now."',
                format='xml|PRODUKTER/PRODUKT/',
                field_merchant='FORHANDLER',
                field_name='PRODUKTNAVN',
                field_description='',
                field_image_url='BILLEDURL',
                field_buy_url='VAREURL',
                field_price='NYPRIS',
                field_category='KATEGORINAVN',
                field_brand='BRAND'
                "
;
      
database_queryModify($sql,$result);
      
admin_import($filename,10);
    }
    print 
"Done.";
    exit();
  }
  print 
"<form method='POST'>";
  print 
"<textarea name='feeds' style='width:100%;height:300px;'></textarea>";
  print 
"<input type='submit' />";
  print 
"</form>";
  require(
"admin_footer.php");
?>

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by macland on Thu, 2020-04-30 10:22

Thank you very much for your quick response :-)

Two (three) problems:

1+2)
I first got an error "str_replace() expects at least 3 parameters, 2 given on bulk_new.php on line 17"

I added a "," and the import then works with a-z and spaces, but feeds with special characters like "æ" gets added to the automation tool - but still with "æ" in the filename so the result on xml-import is "invalid file name"

3) The feeds are not registrered, but that might be some other error, because when I try to do the entire proces manually I end up with the same problem, no error messages, just nothing registrered.
Can it be because the feeds and the database are on different servers?
The cron jobs are added, so there is permission to write to the db

Submitted by support on Thu, 2020-04-30 12:56

Hi,

I've modified the above to correct the str_replace() call (had "." instead of ",") but also to translate all characters in the generated $filename to ASCII with the addition of;

setlocale(LC_ALL,"en_US.utf8");

and

$filename = iconv("UTF-8","ASCII//TRANSLIT",$filename);

I then ran the script through on my test server with the example CSV exactly as posted above and all worked as expected - could you perhaps try that on your server - they will show 0 imported products of course because the example.com server just returns a generic HTML response; if that works but your real data is not if you could perhaps post an example couple of lines (I'll remove before posting your reply) I'll try that out on my test server...

Cheers,
David.
--
PriceTapestry.com

Submitted by macland on Fri, 2020-05-01 08:17

Hi David
Thank you very much.

The two feeds you provided worked fine :-)
For the other feeds some special characters got replaced with ? so it still gives invaled filename error. But for others it work.

You can try fx with

{code saved}

The first and last feed works, but the two in the middle gives an error because of the "ø"

Also don't worry about importing, I've gotten the 24 hour limit removed :-)

In any case I also still can't registrer the feeds, but I think that a completely different problem - I just tried connecting to my old database and here the new feeds are registrered fine, so that's probably a server problem.

Thank you again very much!

Submitted by macland on Fri, 2020-05-01 08:39

Regarding registrering the feeds I've fixed that now - for some reason I had ran the wrong dbmod.php script, adding description to categories instead of the ean field :-)

Now feeds are registrering and products are importing just fine :-)

Submitted by support on Fri, 2020-05-01 09:26

Hi,

Looks like it's nearly there - so just regarding the characters not covered by the iconv translation; if it would be OK for these to be swapped out as underscore which is valid in filename values; where you have the following code at line 19:

      $filename = iconv("UTF-8","ASCII//TRANSLIT",$filename);

...REPLACE with:

      $filename = iconv("UTF-8","ASCII//TRANSLIT",$filename);
      $filename = str_replace("?","_",$filename);

Hope this helps!

Cheers,
David.
--
PriceTapestry.com

Submitted by macland on Fri, 2020-05-01 10:36

Just added 400+ feeds and it works like a charm! Thank you very very much! :-D