Hi,
I am getting a filemtime error on the admin index page but it's only happening with one file. Here's what I have been able to find out so far:
-The modified date shows 1969-12-31 00:00
-The modified date shows the correct date when the file is zipped but when it's unzipped, it changes to the 1969 date.
-If I link to the file directly, I get a 404/403 error, permission denied
-The autodetect script shows up blank and, no matter which option I choose or which magicparser string I enter, the feed always looks blank
This is an EXTREMELY large datafeed, about 2GB.
Any ideas what might be happening here?
David,
Yeah, that was the very first thing I tried. I tried "touch" and "settime" and neither one of those commands fixed the filemtime error. After some further investigation, I found a comment on php.net referencing the exact problem I was facing. It seems that my PHP's integer type is only 32 bits. Rather than trying to change that, I just tried the workaround given in the example.
I basically set the path to my file (since it is the only one I'm using) then used
$mtime = exec ('stat -c %Y '. escapeshellarg ($path));
Then, I changed line 98 of admin/index.php to read
$modified = $mtime
This corrected my filemtime error but did not correct my other problems. I can not register the feed, it still shows blank.
How do I change php's integer type? Maybe that will help.
Or, are there other locations within the pt script that I will need to change to use this workaround.
Thanks for your help, as always,
Mike
Hi Mike,
With a file size of 2GB, it is possible that it is a timeout problem on the first stage of registration - where the feed format is autodetected. When you click "Register" along side this feed, do you immediately receive an error - and if so, what error message to your get; or is there a fixed period, say 30 seconds, and you then get a blank screen?
If you've not already come across the timeout help thread it might be worth a read incase you can identify that as the problem...
http://www.pricetapestry.com/node/582
Even if this is not the case, if you are regularly working with feeds of this size, and you know the format, it may be worth registering them manually without the auto-detection. This is particularly easy if you are using smaller feeds of the same format - just copy the "Format String" from their registration page 1; otherwise you might be able to use the "more" command to work out the format string, and then use scripts/register.php to register the feed. I would also recommend using scripts/import.php from the command line to import such large feeds as this is much less prone to any timeout issues....
http://www.pricetapestry.com/node/6
Cheers,
David.
David,
There is no error message once I click register, it was only the filemtime error message on admin/index.php.
The auto-detect shows up blank, so I tried manually inputing the auto-detect string, the feed still showed up blank.
The next thing I tried was to import a portion of the feed, about 30 lines. I downloaded a portion of the large file, then reuploaded it to my feeds directory. I then, successfully, registered the smaller feed and imported the 30 products.
After that, I copied the format string, for the larger file, just to make sure that I had done it correctly and the fields still showed up blank.
The last thing I tried was to go into PhpMyAdmin and manually enter the data into the feeds table, trying to "force" the script to recognize the feed. I used shell access to try and import the feed and it just gave the message "importing largefeed [done]" and no feed was imported.
I'm sure that this has something to do with the size of the file, the smaller file imported successfully.
A couple links, I'd appreciate it if you "saved" them from public viewing because I've removed the password access for this location temporarily:
{link saved}
itune.txt is the smaller file and 13508_1928044_30_cmp.txt is the larger one. You can see how the larger file shows an imported date, that is because of the "forced" import I tried.
Hi Mike,
It sounds like PHP is having trouble simply opening the file. I would try running the following test script inside your admin/ folder to see if this is the case:
test.php
<?php
$f = fopen("13508_1928044_30_cmp.txt","r");
if ($f)
{
print "fopen success!";
}
else
{
print "fopen fail!";
}
?>
Depending on the outcome of this script we'll know where to carry on looking...
Cheers,
David.
I uploaded it to my feeds directory and got the following error message:
"Warning: fopen(13508_1928044_30_cmp.txt) [function.fopen]: failed to open stream: File too large in /home/xxxxx/public_html/music/feeds/test.php on line 2"
fopen fail!
If I change the filename to itune.txt (the smaller one), I get:
fopen success!
Is this a setting in php that needs to be changed?
Thanks for your help,
Mike
Hi Mike,
At least that proves that it is a fundamental PHP issue, not being able to open the file. I've looked into this, and it seems that the standard PHP build uses a 32bit file pointer, making the maximum file size 2GB. To work around this, you would need to re-complie PHP with 64 bit file pointers by editing Makefile and adding -D_FILE_OFFSET_BITS=64 to the CFLAGS line, and then re-compiling....
Hopefully that's an option for you. If not, it might be worth contacting the merchant to see if a smaller
Hi Mike,
It does sound like the unzipped file is causing a problem for your file system. The modified date / time for an unzipped file is quite often the timestamp of the original file as it is preserved by the .zip archive, so the actual modified time shouldn't matter.
However, it may be highlighting some strange OS bug, so my first thought would be to update the timestamp, although i'm not quite sure about the best way to do this for such a large file. If you have shell (Telnet or SSH) access to your account, you could use the touch command, for example (where $ is the command prompt):
$cd path/to/pricetapestry/feeds$touch verybigfile.xml
Hope this helps,
Cheers,
David.