forum.coppermine-gallery.net
No Support => Modifications/Add-Ons/Hacks => Mods: Uploading => Topic started by: Nibbler on January 21, 2005, 11:37:57 am
-
Here we go, upload this into your coppermine directory and call it timer.php (I hardcoded it, so sue me :P) then go there, you know the drill.
It will keep reloading until all the pics are added if you set it to, so you can start it and leave it unattended for hours until it finishes, even with php set with a very low max_execution_time (ignore the fatal error). Not recommended for shared hosting environments unless you are vewy vewy careful. :)
Note: Auto reload needs js support.
[Note: Tagged as 1.4, see later posts in thread for updated versions of this mod]
-
You, sir, are a legend.
;D
I thank you for your sterling work.
I can't express how awesomely happy I am with this hack! :D :D :D
edit (heh..)
EXCEPT... it's a shame it can't resume a folder/cat... if I bung a load more files into the same folder then ask your amazing script to do its work again, it starts a new catagory and populates it with new albums, all of the same names as before, and starts from scratch.
Shame about that :(
-
Thank you so much Nibbler,
So glad to be able to batch input more than 1 album at a time.
I can't get to have the autorun / auto reload working eventhough JS is definitely on.
I tried both on Mozilla Firefox and IE...
Any ideas?
Thanks,
Thomas
-
I get the following error when I try to run this:
"Created root category
Warning: Invalid argument supplied for foreach() in /var/www/html/coppermine/timer.php on line 200
structure created, 0 files to add."
The category is created, but no albums or pictures are included. (I've gotten more complicated errors before, but I got the above trying to use it add a single image, in a single directory)
I.e. directiory albums/test contains test.jpg and nothing else. permissions are all 775.
Any suggestions as to what might cause this?
Thanks,
Orie
-
OK, you make a folder in albums -> that becomes the root category.
You cannot then have any pics directly in that folder since you cannot have pics directly in a category.
You need to make folders inside the root folder - these become your albums, or more categories.
Bottom line is - You can only create a directory/file structure that would be a legal category/album/pic structure in coppermine.
-
Ahh, I think I understand it. I need a folder structure that looks like:
Category/Album/Image.jpg right?
I redid my test and it worked; although it still spit out an error, it did insert my test image as it should.
Edit, it works with one image, but when I have more than one image I get this following error:
"Fatal error: Call to undefined function: array_diff_assoc() in /var/www/html/coppermine/timer.php on line 98
structure created, 700 files to add."
Thanks for your help.
Orie
-
Ah, seems the script requires PHP 4.3.0 or better to work. You can try adding the substitute function in the first comment here (http://uk.php.net/function.array_diff_assoc) into the script, or you could upgrade your PHP, or you could try using PHP_Compat (http://pear.php.net/package/PHP_Compat).
-
Nibbler - Great script.
"It needed more power; so I rewired it" (Tool Time Joke, hope somebody gets it)
I modified the script so that #1 you can run it against the album directory itself as the root cat.. There's an exclusion list that eliminates the basic garbage folders you might find there.
#2 It eliminates trying to add thumb_ and normal_ images that are already there from previous batch attempts.
#3 Checks to see if it can use a pre-existing cat/album before creating the new ones.
Which allows you to run this to do your batch adds all the time, successive runs don't hurt anything.
I was using this script against 1.4, but I didn't change anything that would be materially different than 1.3 so I would expect it to work on both platforms.
Also a nice little side effect of running this with 1.4 is that IPTC titles,keywords, and comments are automatically imported if you have iptc turned on in the config.
-
@mrplow With my changes there should be no problem resuming. Do it as many times as you like.
-
Edit (double Post)
......
-
Hello
With both Scripts i get the following Error when i call the timer.php from my Browser:
Parse error: parse error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING in /abc/def//timer.php on line 174
Any hints?
Thank you...
-
im sure the line number that errors out isn't the same between the two scripts. make a note of which script gives which line number.
-
This is probably a problem with the file format. I've attached a plain text version of Donnoman's file. Try this one and see if you still get those errors.
-
Hi Kegobeer
Thank you for the Script. With this one i recieve the following Error:
Parse error: parse error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING in /abc/def/timer.php on line 416
Any hints again?
Junior
-
That file is 266 lines long. Therefore there cannot be an error on line 416. I suggest you try a different ftp client.
-
Hi
Iam sorry. I fixed the Script. Now its exactly 266 lines long and my Error-Msg is:
Parse error: parse error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING in /abc/def/timer.php on line 192
Line 192:
$auto = (isset($_POST['auto']) && $_POST['auto']) ? 'checked = "checked"' : '';
???
-
Are you uploading your files as ASCII or binary? These files need to be uploaded as ASCII, or use the auto switch so your FTP client will do it for you.
-
I uploaded the filles in ASCII of corse ;D
-
Then perhaps you should try a different FTP client.
-
Hi,
here's my small participation to this nice and very usefull script..
Unfortunately as i didn't read the whole topic and especially donnoman's posts i didn't see that some of my modificatons had already been introduced and i just grabbed the file at the top of the topic, expecting it to be the latest version.. i was quite in a rush and started to make my modifications right away, silly me :)
Here's how the script works :
The script expects a directory name under ./albums/, that directory is supposed to be a root category in your gallery. Bottom directories, that should contain your pictures, will be treated as albums and intermediate directories will be treated as sub categories from the root category of course..
here's a quick example, let's say you have the following root category tree :
./albums/lib/
you should declare "lib" as the root directory to process.
now if you have the following tree under lib :
./albums/lib/A/Aname/pic.jpg
a root category named "lib" will be created, a sub category "lib>A", name "A" will be created, an album "lib>A>Aname" named "Aname" will be created and the picture pic.jpg will be added to the album "lib>A>Aname"..
that's how the original script works..
Now here are the features i've added to the original script :
In the same way donnoman did it, i added some filtering in order to avoid adding the same category/album/picture twice. So if you run the script twice on the aboce dirctory tree, everything should be created/added the first time and nothing will be done the second time..
Beware that there is a "limitation" in my database query on how category and album names are matched. Indeed a side efect of the query will be to limit creation of categories and albums to unique names. Which is not a limitation in the orignal design of CPG but is obviously a limitation of the filesystem (you cannot have the same directory name twice at the same location).. So you cannot have two categories or albums with the same name at the same location.. For example if you manually create, thru cpg interface, a second "Aname" album under "lib>A" and run the script to add new pics located in "/lib/A/Aname/" the script will use the first ID of the albums named "Aname".. maybe it's not worth mentioning, i don't know.. The same thing will happen with Category Names..
Of course if your second "Aname" album is located under "lib>Amisc>" then it's not a problem as the script will always use the reference to the parent when it tries to find if an album or category name already exists.. usually different parents mean different children ;)
Then the pictures filter was already there in the original script but was applied while processinf every pictures found.. I've add to that filter a global pictures filtering that will remove from the pictures lists build on the directory tree, the pictures already inserted in the database.. So for example if you update the directory tree under"./albums/lib", let's say it contains 20k+ pics but only have 150 new pictures, then my script will purge the already added 20k+ pics and add directly the 150 new pics.. quite nice isn'it :)
A thing you shoul also know it that i've modified the way filenames are build.. In the original script any char not being part of the lists a-z A-Z 0-9 would be replaced by the char "_" (underscore).. I've improved that by removing existing double underscores "__" until there aren't any doubles left..
You'll probably notice that the renaming process will crush any non US chars, so you might want to modify that if you filenames have international chars in it..
Well i think it's all i can say about it..
I've tested it against a quite huge fake directory tree made of 26 subcategories (a-z) and 26x26 albums (26 albums in each 26 subcategories), each album containing 50 times the same picture named from 1.jpg to 50.jpg, which make a total of 33800 pictures..
The script take a few seconds (25sec) to build and filter the directory tree, which is located on a remote nfs server, then it will process the pictures according to the reloading parameters you've chosen.. The first update took a few hours as expected.. Then adding a few pics here and there just take a few seconds, no need to go over the previously added 30k pics right ? ;)
Thx for the coder of the original script..
Enjoy,
flux
-
i get the following error message:
Warning: array_diff_assoc(): Argument #2 is not an array in C:\WebRoot\photogallery\timer.php on line 102
After filtering 0 pictures will be added
structure created, 0 files to add.
im trying to use flux's versio... plz help!! :(
the original one by nibbler works great though but the other 2 modified versions are giving me errors. it'd be nice to have the extra features. any help is appreciated. thanks!
-
The idea for the timer script is what i have been searchin now for almost a year for my CPG, unfortunatly none of the three works for me.
Since I was feed up with the normal upload process my gallery stopped at 615 Pics -> although on my local disc i got 3000+. So now i did as discribed set up a directory structure under albums/test/test/ and uploaded some random pics for a quick test - but all the three tested scripts give me the following back "Using root Category
structure created, 0 files to add." -> But there is no structure created.
So schould i use a fresh install of CPG for the scripts or are there any special options in CPG which i schouldn't use?
Thx for the script anyway hope it works for me too soon
Setup:
Apache 1.xxx
PHP 4.3.8
-
Make sure you setup a legal file structure - check the previous postingt about this.
-
thx for the tip
my directory structure looks like this (nevermind the word since its german)
/albums/Schloss/Sommer/xxxxxx.jpg
So is this legal or do i have to change something?
Solved it: installed cpg1.4.1 now the script isrunning flawless
-
Ok maybe my flawless was to fast written ;D
After a night of running, i counted 2549 files in 53 albums and 28 categories added. When comparing the structured in CPG with the directory structure in /albums. I found one directory which wasn't added, i'm not speaking of userpics and edit, I mean a drectory with a legal structure.
the structure would be: /albums/Urlaube & Reisen/Wagrain Skifahren/Februar03
when running again found out the, creation process for albums and categorys stops and displays a warning message and stops. Pls check attached picture for the look of the message.
any suggestions??
-
The '&' in the folder name may be causing the issue. Try using something else.
-
The & can't be the error because it is in another categoryname too and that one was addede correctly.
-
So enable debug mode and post the actual mysql error message ;)
-
thx for the debug tip, solved the problem. The error was in another directory with an album name with an invalid character.
-
Thanks to all who've done a great job on this script. I started with a fresh install of Coppermine and the scripts would create the categories, but not populate the albums. I realized that the scripts fail if there are no pictures already defined in the database!
So import a single file by hand to ensure that at least one picture is in the gallery. Then proceed with the timer script!
Thanks,
Mark
-
works great. Installed from scratch and have uploaded over 1,000 pics with no problems. Saved me soo much gime.
www.razorbacklegacy.com/hog_archive (http://www.razorbacklegacy.com/hog_archive)
-
I'm having problems with the original script. It doesn't upload the pictures, just creates category and albums.
My dirs look like this: /var/www/localhost/htdocs/fester/albums/Categories/Albums/Pics.jpg <- thats okey right? It's all chmodded to 777.
This is what i get when i click begin in the script:.
Created root category
created album 000429_Valborg
created album 000517_Boras
created album 001012_Forfest_Kaneback
created album 001105_Kanarieoarna
created album 000101_Fest_Atvid
created album 000101_Pokerkvall_David
created album 000205_Pokerkvall_Bigalow
Notice: Undefined index: auto in /var/www/localhost/htdocs/fester/timer.php on line 175
structure created, 325 files to add.
When i click continue to convert the pictures nothing happens.
Here is some info:
USER:
------------------
Array
(
[ID] => 826d834b7f0764b5223187fec79ce0d4
[am] => 1
[liv] => Array
(
[0] => 236
[1] => 176
[2] => 229
[3] => 223
[4] => 209
)
)
==========================
USER DATA:
------------------
Array
(
[user_id] => 1
[user_group] => 1
[user_active] => YES
[user_name] => niuck
[user_password] => ********
[user_lastvisit] => 2005-07-01 18:57:34
[user_regdate] => 2005-06-29 05:51:31
[user_group_list] =>
[user_email] =>
[user_website] =>
[user_location] =>
[user_interests] =>
[user_occupation] =>
[user_actkey] =>
[disk_max] => 0
[disk_min] => 0
[can_rate_pictures] => 1
[can_send_ecards] => 1
[ufc_max] => 3
[ufc_min] => 3
[custom_user_upload] => 0
[num_file_upload] => 5
[num_URI_upload] => 3
[can_post_comments] => 1
[can_upload_pictures] => 1
[can_create_albums] => 1
[has_admin_access] => 1
[pub_upl_need_approval] => 0
[priv_upl_need_approval] => 0
[group_name] => Administrators
[upload_form_config] => 3
[group_quota] => 0
[can_see_all_albums] => 1
[group_id] => 1
[groups] => Array
(
[1] => 1
)
)
==========================
Queries:
------------------
Array
(
[0] => SELECT extension, mime, content FROM cpg133_filetypes;
[1] => SELECT * FROM cpg133_users WHERE user_id='1'AND user_active = 'YES' AND user_password != '' AND BINARY MD5(user_password) = '48f8d008981572fd9fc3218c2460c833'
[2] => SELECT MAX(group_quota) as disk_max, MIN(group_quota) as disk_min, MAX(can_rate_pictures) as can_rate_pictures, MAX(can_send_ecards) as can_send_ecards, MAX(upload_form_config) as ufc_max, MIN(upload_form_config) as ufc_min, MAX(custom_user_upload) as custom_user_upload, MAX(num_file_upload) as num_file_upload, MAX(num_URI_upload) as num_URI_upload, MAX(can_post_comments) as can_post_comments, MAX(can_upload_pictures) as can_upload_pictures, MAX(can_create_albums) as can_create_albums, MAX(has_admin_access) as has_admin_access, MIN(pub_upl_need_approval) as pub_upl_need_approval, MIN( priv_upl_need_approval) as priv_upl_need_approval FROM cpg133_usergroups WHERE group_id in (1)
[3] => SELECT group_name FROM cpg133_usergroups WHERE group_id= 1
[4] => DELETE FROM cpg133_banned WHERE expiry < '2005-07-01 19:32:34'
[5] => SELECT * FROM cpg133_banned WHERE ip_addr='192.168.1.1' OR ip_addr='192.168.1.1' OR user_id=1
[6] => SELECT parent FROM cpg133_categories WHERE cid = '33' LIMIT 1
[7] => UPDATE cpg133_categories SET parent='0' WHERE parent = '33'
[8] => UPDATE cpg133_albums SET category='0' WHERE category = '33'
[9] => DELETE FROM cpg133_categories WHERE cid='33' LIMIT 1
[10] => SELECT cid FROM cpg133_categories WHERE 1
[11] => UPDATE cpg133_categories SET parent = '0' WHERE parent=cid OR parent NOT IN (1)
[12] => SELECT cid, name, description FROM cpg133_categories WHERE parent = '0' ORDER BY pos
[13] => SELECT cid, name, description FROM cpg133_categories WHERE parent = '1' ORDER BY pos
[14] => UPDATE cpg133_categories SET pos='0' WHERE cid = '1' LIMIT 1
[15] => SELECT pid, filepath, filename, url_prefix FROM cpg133_pictures,cpg133_albums WHERE cpg133_pictures.aid = cpg133_albums.aid AND cpg133_albums.category = '33' AND approved='YES' ORDER BY filename
)
==========================
GET :
------------------
Array
(
[op] => deletecat
[cid] => 33
)
==========================
POST :
------------------
Array
(
)
==========================
VERSION INFO :
------------------
PHP version: 4.3.11 - OK
------------------
mySQL version: 4.0.24
------------------
Coppermine version: 1.3.3
==========================
Module: gd
------------------
GD Support enabled
GD Version bundled (2.0.28 compatible)
GIF Read Support enabled
GIF Create Support enabled
JPG Support enabled
PNG Support enabled
WBMP Support enabled
XBM Support enabled
==========================
Module: mysql
------------------
Active Persistent Links 0
Active Links 1
Client API version 4.0.24
MYSQL_MODULE_TYPE external
MYSQL_SOCKET /var/run/mysqld/mysqld.sock
MYSQL_INCLUDE -I/usr/include/mysql
MYSQL_LIBS -L/usr/lib -lmysqlclient
==========================
Module: zlib
------------------
ZLib Support enabled
Compiled Version 1.2.2
Linked Version 1.2.2
==========================
Server restrictions (safe mode)?
------------------
Directive | Local Value | Master Value
safe_mode | Off | Off
safe_mode_exec_dir | no value | no value
safe_mode_gid | Off | Off
safe_mode_include_dir | no value | no value
safe_mode_exec_dir | no value | no value
sql.safe_mode | Off | Off
disable_functions | no value | no value
file_uploads | On | On
include_path | .:/usr/lib/php | .:/usr/lib/php
open_basedir | no value | no value
==========================
email
------------------
Directive | Local Value | Master Value
sendmail_from | no value | no value
sendmail_path | /usr/sbin/sendmail -t -i | /usr/sbin/sendmail -t -i
SMTP | localhost | localhost
smtp_port | 25 | 25
==========================
Size and Time
------------------
Directive | Local Value | Master Value
max_execution_time | 999999 | 999999
max_input_time | 300 | 300
upload_max_filesize | 2M | 2M
post_max_size | 16M | 16M
==========================
Page generated in 0.237 seconds - 16 queries in 0.012 seconds - Album set :
Anyone knows what could be wrong?
I'm using 1.3.3 in Gentoo if that matters.
Thanks!
-
Hi there !
I'm having some troubles with timer.php.
Well, I'm using Coppermine 1.3.3 , Apache 2, PHP 5 on a FC4 distro.
When I try to use timer.php, I get this Fatal error :
"Fatal error: Call to undefined function cpg_db_query() in /var/www/html/cpg/timer.php on line 241"
My "albums" directory's structured as following :
albums
FTP
categ
album
.jpg files
After a litlle search, I don't find any cpg_db_query declaration ... Is this hack working with cpg 1.3.3 ? Am I missing something ?
Thanks for your help !
-
Looks like you are using a version designed for cpg1.4. Which version of the mod are you using ?
-
I've tried the one post by Flux, and the mod posted by donnoman (your mod modified to enable retries).
In some posts, I've seen some answer for similar problems on several mods (between 1.4 and 1.3.x), and the solution was to replace cpg_db_xxx function call by db_xxx. Is that a solution for me too ?
!!! EDIT :
I've solved my problem : I've changed changing cpg_db_query function call to db_query which is the correct function nam in cpg 1.3.3.
My mod was effectively for cpg1.4, sorry for my mistake Nibbler (and for my poor english of french guy) ;D
-
I took Flux's timer.php and merged a couple of my changes to his code. This is tested with cpg 1.3, for 1.4 you should just need to change the db_query's with cpg_db_query's.
-
lo,
@donnoman
what kind of changes did you add to my script exactly ?
i was stupid enough not to see that a more advanced script than the original one had been already made when i started mine..
basically the only thing i made in my script that may not have been in the previous script was to skip all the already existing pictures in the galleries at once, that increased the processing speed greatly.. besides that my script just made sure categories/albums didn't already exist before creating new ones, as the other script does..
And yes i made my script for 1.3 only so it shouldn't work with 1.4 and to make it work you only have to replace the db function name..
-
I liked your method of filtering out stuff that already had been imported. So I dropped most of my code in favor of yours. I just added the pieces that allow you to pass the intial directory as null and it will import everything under the "albums" directory.
-
Hi,
I have some problem by adding with the postet scripts (i tryed all, from flux&donnoman). I must add 38 Gigabyte Pictures (50k+) and some pictures have a size of >5mb, all pictures under 1mb add the script without problems, when a bigger picture in the directory the whole script stop..
i have changes the php.ini of 2400 and 500mb, and have in the config from coppermine 6000kb, and the script can´t import a file with 1500kb, why this?
please help me.. pascal
-
don't think it has anything to do with the scripts..
as far as i can tell you've covered with your settings the various sources of limitations (cpg & php)..
Now if you don't get any debug info it'll be hard to debug, i guess you should turn on dubug in cpg config and maybe you'll get some interesting output when the script fails..
But personnaly i'd put my money on the image component, not sure GD2 or image magic are bug free..
Once you've found some debug info you should ask some support in a support forum..
A good test would also be to try it thru cpg integrated batch add tool and see what happens, if it fails too then it has nothing to do with the script itself but more with a cpg core funtion maybe or with a server component (gd, php, apache and so on..) and don't forget to turn debug on in your cpg config..
-
I just added the pieces that allow you to pass the intial directory as null and it will import everything under the "albums" directory.
Hello.
Nice script.
It works well, imports everything under /albums. But doing so with a null value to 'SubDirectory (under albums)' results in
'filepath' in 'cpg***_pictures' starting with a slash.
If the picture is already there, by other methods: it will be inserted twice.
It produce adresses like http://**/cpg133/albums//***/**/*.jpg , double slash after albums.
Any way of filtering out things like '../../../..' as possible value 'SubDirectory (under albums)' ?
Mvh Andrez1
-
i haven't analyzed how the directory structure is created but a quick and dirty way to solve the problem would be to simple remove a possible "/" first char in the path string just before insterting the picture..
i guess that having an openning "/" in the filepath data is not correct so the following simple patch would be okay.. although the proper thing to do would be to sanitize the file path when building the directory structure..
So using the latest Donoman file, below line 178 :
$dir_name = dirname($pic_file) . "/";
insert :
$dir_name = ( substr($dir_name,0,1) == "/" ) ? substr($dir_name,1) : $dir_name;
that should do it. If so please confirm it works fine, i have nothing to test it on here, the code works don't worry, just make sure it also works when using a subcategory info in the field..
then i'll post the updated file.
thx for your bug report
.flux
-
So using the latest Donoman file, below line 178 :
$dir_name = dirname($pic_file) . "/";
insert :
$dir_name = ( substr($dir_name,0,1) == "/" ) ? substr($dir_name,1) : $dir_name;
that should do it. If so please confirm it works fine, i have nothing to test it on here, the code works don't worry, just make sure it also works when using a subcategory info in the field..
then i'll post the updated file.
That makes the script take a blank value.
It also works when giving as a value a subdir containing another dir with imagefiles.
although the proper thing to do would be to sanitize the file path when building the directory structure..
Absolutely..
Tested a litle bit.
the file (win):
c:\XXXXXX\htdocs\cpg133\albums\C370-3\TJA14\P8190003.JPG
in subdir C370-3\TJA14 is accepted at the following values:
(aid,filepath,filename) (given value to the script)
6236,\\c370-3\\/tja14/,P8190003.JPG \c370-3\
6233,c370-3\\/tja14/,P8190003.JPG c370-3\
6230,c370-3//tja14/,P8190003.JPG c370-3/
6227,c370-3/tja14/,P8190003.JPG c370-3
6224,\\c370-3//tja14/,P8190003.JPG \c370-3/
The values given to the script results in 5 categories like the values given to the script. And a 'tja14' album in each.
The 'c370-3' value seems to be what the script is expecting. But it accepts a lot (including '..', which might be a risk).
Mvh Andrez1
-
That makes the script take a blank value.
But filtering on what is already in the database is broken.. That does not work with the newly added line to the script, and a null value passed to the scripts 'SubDirectory (under albums)'.
Filtering does work when the script is given a subdir as value.
Mvh Andrez1
-
okay,
based on your comments i've made a few modifications to the latest version of donnoman's script.
it will now handle blank "subcategory" field properly and store valid paths in the database, without starting "/".
Filtering also works with blank "subcategory" field.
Concerning your previously insterted pictures using a blank "subcategory" field they were all stored with a faulty filepath so you'll have to fix them. The following sql query will do that for you :
UPDATE prefix_pictures SET filepath=SUBSTRING(filepath,2) WHERE RIGHT(filepath,1)='/'
of course replace "prefix_pictures" table name by your own table name using your "prefix_" name..
that's about it..
oh yeah i've also added some more version checking in the script so it'll fail to start if you're not using cpg v1.3.x or 1.4.x
And it'll also work now for both versions 1.3.x and 1.4.x, i've just wrapped the db_query function BUT did some tests on a cpg1.3.5 version only !
So if you can confirm it works fine on a 1.4.1 beta too, i'll thank you for that ;)
.flux
-
one last quick word, all my tests are done on a linux plateform so if there are some paths problems when running this script on windows, i'm afraid i won't be able to do much about it.
-
it will now handle blank "subcategory" field properly and store valid paths in the database, without starting "/".
Filtering also works with blank "subcategory" field.
Works fine in cpg 1.33 an 1.41.
The 1.33 also gets filename as title.
The 1.41 do not getfilname as title. (and thats ok, can be set in config on thumbs)
Have not used the script for anything serious. Yet. Nothing to repair.
As of now basic functionality is there, working. Fine to insert large amounts of pictures.
I have found 3000 pictures in a directory, presized and with thumbs_ present, to be the practical limit with 'batch add' on a local pentium4 - 3ghz -xp. (and that is with standard values on script time in php). More than that and 'reload' or new start on given directory is needed.
The 'batch add' gives possibility to set which album to insert pictures into. This script does not. But it solves a problem with inserting large amounts of pictures. That is the use i consider. And when it's done something like -
UPDATE `cpg132_pictures` SET `aid` = 3 WHERE `filename` between 'xxx005.jpg' and 'xxx010.jpg'
(or whatever the WHERE clause and 'aid' is set to) - will arrange the pictures in another order than the directory-structure.
(Hiding the albums originating from the script is part of idea.)
The next time the script is run on the given directory it does not find the picture in db and do not filter it out. It tries to add it and reports something like 'picture already there'. Thats OK, but not optimal.
Mvh Andrez1
-
one last quick word, all my tests are done on a linux plateform so if there are some paths problems when running this script on windows, i'm afraid i won't be able to do much about it.
Not a windows-linux path problem. '..' is understood as 'one directory up' in windows, linux and html.
'../themes' as a value to the scripts 'SubDirectory (under albums)' gives a result i think not shold be allowed.
'/dir' or '/dir/' or 'dir/subdir' (given existing dirs) produces categories and albums and insert files into them. I think it should not.
I have a category 'c370-7/7' with the album 'tja22' an a file like 'P8250003.JPG'
http://XXXX/cpg133/albums/c370-7/7/tja22/P8250003.JPG
That is produced by passing 'c370-7/7' to the script. (dir and subdirs exist, there is nothing in the script to guide me, stop me or warn me.)
I also have a 'c370-7' category with a '7' subcategory and 'tja22' album. Thats empty. Should not be.
And so on. Gets messy.
Mvh Andrez1
-
not sure i understand why your previously added pictures are not filtered but then skipped with an "already added" output.. shouldn't happen, something must be broken with the filtering, although what i've tested here was working fine as far as i could tell..
as for "../" it's not supported and not filtered either because it's simply a bad use of the subdirectory field..
The script and the whole cpg system only deals with pictures placed in your "album" (aka fullpath in the config) directory directly under the root directory..
I guess it must have been stated somewhere in the first explanations of this batch add tool that the subdirectory field was made to target a directory under the "album" directory..
thus using "../" entries makes no sence as it would mean leaving the "album" directory tree..
that's all i can say..
-
not sure i understand why your previously added pictures are not filtered but then skipped with an "already added" output.. shouldn't happen, something must be broken with the filtering, although what i've tested here was working fine as far as i could tell..
That happens on a cpg133 when pictures are inserted using the script and a value to 'SubDirectory (under albums)' (f.ex 'c370-8')
The dir 'c370-8' exist and have a subdir 'tja23' which contains pictures.
'http://xxxxx/cpg133/albums/c370-8/TJA23/normal_P8280001.JPG' is visible in coppermine after use of the script. The image is found in the category 'c370-8', album 'tja23'. It is as it should be.
But there is no need to stick to the somewhat rigid order imposed by the script.
The mentioned album 'tja23' have an 'aid' like '11250' , and a new album created like 'fancyname' with an 'aid' like '11251'.
Running --- UPDATE `cpg133_pictures` SET `aid` = 11251 WHERE `aid` like '11250' --- on the db empties the album created by the script and move the pictures to another album. The album 'fancyname'. That kind of use is neccesary if someone want another name on an album other than the subdir the file resides in. That kind of use make the script useable.
The only thing changed is the 'aid' field in `cpg133_pictures`.
the pair of 'filepath' and 'filename' is the same.
as for "../" it's not supported and not filtered either because it's simply a bad use of the subdirectory field..
The script and the whole cpg system only deals with pictures placed in your "album" (aka fullpath in the config) directory directly under the root directory..
I guess it must have been stated somewhere in the first explanations of this batch add tool that the subdirectory field was made to target a directory under the "album" directory..
thus using "../" entries makes no sence as it would mean leaving the "album" directory tree..
It can be a feature ;) to be allowed to list the whole directorystructure of the disk (filesystem?) on a server running coppermine.
'../../..' to the script gives me a category '../../..' with a subcategory 'htdocs'.
'../../coppermine/albums/test1' to the script gives me a category '../../coppermine/albums/test1' containing albums. From another coppermine-install that is..
If I ask for image properties on:
'http://xxxxx/cpg133/displayimage.php?album=7182&pos=0'
I get
'http://xxxxx/coppermine/albums/test1/alt/xxx001.jpg'
'filepath' on that picture is set to '../../coppermine/albums/test1/alt/'
A nice feature which can be exploited.
Mvh Andrez1
-
Should the image next to the thread say "1.4.x" as well? Maybe the most current version should be added to the first message (with Nibbler's permission & an edit note of course)?
The current version is here:
http://forum.coppermine-gallery.net/index.php?topic=13949.msg105809#msg105809
I have some mods I'd like to suggest, like ignoring the "Thumbs.db" files Windows creates, a test run that doesn't create albums (to check things like when you rename an album in CPG and then timer.php wants to add a new album with the current folder name but now old album name - a mod I'm trying to decide how to address), and a mod to remove forbidden characters in directory names (like CPG does with filenames). I'll add them here when I get the chance. I'd also like to try to convert this into a plugin. Has anyone attempted this yet? Of course it needs to be more bulletproof and have more checks installed.
-
I'd be happy to host this project over on cpg-contrib.org, and we could have it made as a plugin, and available from cvs.
Perhaps a better name than TIMER might be appropriate ;)
When someone figures out a good name, I'll create the project.
-
if its not too much to ask, would it be possible to get this nicely made timer script to include the mini thumbs and orgi back-up pix? im using Stramm's mod pack found here, http://forum.coppermine-gallery.net/index.php?topic=24341.0 and his mods add two additional files for each pix... one with the prefix mini_ and another with orig_ . And when I run the most recent timer script by flux, it detects these additional files as a completely new pictures and starts creating normal_ and thumb_ files for them. So would it be possible to get the script to include or perhaps simply just ignore the orig_ and mini_ files? If it involves alot of scripting, then no worries.. ill just stop using mini thumbs and watermarking. thanks!
-
So would it be possible to get the script to include or perhaps simply just ignore the orig_ and mini_ files?
I haven't used Stramm's mod pack, but I did modify the timer script to ignore the "Thumbs.db" files that Windows creates when you browse Thumbnails in a directory. The script above already ignores the thumb_ and normal_ files that Coppermine creates, so you merely have to add orig_ and mini_ to the 2 if statements that do this.
In the createstructure function, look for lines similar to these and modify as shown:
foreach ($contents as $file) {
if ( (strncmp($file,$CONFIG['thumb_pfx'],strlen($CONFIG['thumb_pfx'])) != 0 ) && ( strncmp($file,$CONFIG['normal_pfx'],strlen($CONFIG['normal_pfx'])) != 0) && ( strncmp($file,'orig_',strlen('orig_')) != 0) && (strncmp($file,'mini_',strlen('mini_')) != 0)) {
$filelist["$path/$name/$file"] = $aid;
}
}
Then in the addpic function, modify the if statement similarly:
if ( (strncmp($file_name,$CONFIG['thumb_pfx'],strlen($CONFIG['thumb_pfx'])) != 0 ) && ( strncmp($file_name,$CONFIG['normal_pfx'],strlen($CONFIG['normal_pfx'])) != 0) && ( strncmp($file_name,'orig_',strlen('orig_')) != 0) && (strncmp($file_name,'mini_',strlen('mini_')) != 0)) {
-
Thanks! Works like a charm. :)
-
Hi guys, I'm have some problems with this - probably user error...
so far I've grabbed the file attached to the first post in this thread (updated Jan 4th 06)
copied it to my coppermine folder as timer.php
when I run it in a browser, I get a load of garbled PHP & a few boxes/fields to complete - nedless to say, it does nothing.
I've uploaded some test files in the format:
FTP_Upload
Test
10_01_2005
11_01_2005
etc
the intention is for them to have the nesc catagory created & them to be posted in the Test Album.
Any suggestions or quick pointers?
A
-
instead of grabbing the first post, why dont you go through the thread and grab the LAST updated copy, as most of the bugs and 1.3 and 1.4 compatability was worked out in the later versions.
-
Okay, I've grabbed the most recent one from this thread & I get the following when Debug is turned on:
While executing query "INSERT INTO cpg143_categories (pos, parent, name) VALUES ('10000', '0', 'Test')" on 0
mySQL error: Field 'description' doesn't have a default value
-
sounds like your on Mysql 5, search the board, it's been discussed before.
-
I was getting the same error as fade2grey (CPG 1.4.3, Win2K3, PHP 5, MySQL 5) and, at the advice of my webhost, I:
* Booted up phpMyAdmin
* Found the table "cpg143_categories"
* Set the DEFAULT column for "description" to NULL.
Ran the newest version of timer, again.
Now, PROVIDED:
* I had a folder under "albums" called "testy" and
* A folder under "testy" called "testy2" (so the directory structure looks like: <root>\cpg143\albums\testy\testy2")
The TIMER script would create the category "testy" with an album in it called "testy2".
That's the good. The bad is it wouldn't upload the pictures in "testy2".
When I turned the debug on, I'd get the following error:
While executing query "INSERT INTO cpg143_pictures (aid, filepath, filename, filesize, total_filesize, pwidth, pheight, ctime, owner_id, owner_name, title, caption, keywords, approved, user1, user2, user3, user4, pic_raw_ip, pic_hdr_ip, position) VALUES ('12', 'testy/testy2/', 'a.jpg', '14702', '17383', '362', '295', '1137635330', '1', '[my server name, edited out]','', '', '', 'YES', '', '', '', '', '', '', 'a.jpg')" on 0
mySQL error: Out of range value adjusted for column 'position' at row 1
File: D:\Inetpub\wwwroot\[my server]\cpg143\include\functions.inc.php - Line: 248
============
Notices:
\include\themes.inc.php
* Warning line 1167: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)
* Warning line 1168: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)
\include\functions.inc.php
* Warning line 162: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)
Little file named "a.jpg". All of 12K.
I tried editing out the "pos" statements in timer.php and then deleting them altogether. I tried changing the DEFAULT column to null and not null or other numbers in the cpg143_pictures table with phpMyAdmin and nothing helped. I tried combinations of the above, too.
Any suggestions? (My webhost does not wish to downgrade MySQL to 4.x.)
-
Something's weird. The error is being caused because the field position is a number field and cannot be assigned to a string like 'a.jpg'. I checked the timer.php script and it has an error in that it sends the filename where the position should be because it has an extra parameter in the function call. But it works fine for me. Maybe MySQL 4 will realize you are sending a string and ignores it whereas MySQL 5 won't let you do this and gives the error. Or maybe both servers have a strict setting and mine has it off whereas yours has it on.
In any case, look for this line in timer.php:
if (add_picture($aid, $dir_name, $sane_name, $file_name)) {
and replace it with this:
if (add_picture($aid, $dir_name, $sane_name)) {
Let us know if it works. Since I don't have this issue, I cannot easily test it. I assume the error in timer.php is due to an earlier prototype for the add_picture function.
-
Paver deserves WAY more karma points.
That worked! Once again, using the directory structure I mentioned above, it created the album and category, but this time it uploaded my file and added it to the CPG database. No errors.
I do want to test further, because the script didn't (earlier, before Paver's changes) skip already added files/galleries, as "was advertised" -- and I want to test with lots of JPEG's -- but I think this is probably gonna work pretty darn well.
I'll keep y'all updated on the progress.
(Also remember, if you're trying this on CPG143, that you have to change the lines that say "db_query" to "cpg143_db_query". That's on pg. 2 of this thread.)
-
(Also remember, if you're trying this on CPG143, that you have to change the lines that say "db_query" to "cpg143_db_query". That's on pg. 2 of this thread.)
Actually, it's cpg_db_query. But the best thing to do is to download the script that's compatible with both 1.3 & 1.4 that's somewhere in this thread. The flux_donnoman mod timer script. I don't feel like looking for it now. That's the one I've been using.
I added a "check files only" option so that I can do a trial run that won't add anything. I always use that before running a real run of the timer script. That way I'm sure I didn't add an extra folder here or there or have a filename with quotes etc. As I mentioned earlier in this thread, I'd like to contribute my additions but I think it's time to make this into a plugin and I haven't quite done that yet - that's when I plan to submit them. Others are welcome to steal my thunder, or we'll see when I get to it.
-
I went back and got the Flux-Donnoman one instead of the one in the first post in this thread (which says it's the newest) and updated it as you describe. Works just great -- and I tried a couple hundred pictures and categories.
The only thing I did have to do was ask my webhost to increase the timeout in PHP.INI.
Now I'm off to search if there's a plugin that improves the response of catmgr.php .... ;D
-
Hi. I love the idea of this mod. :P
Has anyone got it working with a local windows installation, (win2k)?
I am logged on through http://localhost/, and when I do http://localhost/coppermine-install-directory/timer.php I just get the standard 'error while processing..' error which (in debug mode) gives an initial message of:
While executing query "INSERT INTO copperminecategories (pos, parent, name) VALUES ('10000', '0', 'personal')" on 0
mySQL error: Field 'description' doesn't have a default value
I do run MYSQL5, but I have followed the instructions above already in the hope that it solves it. Also, I have set the folder I made under /albums as writeable by everyone.
Thanks (in advance).
-
Did you try ...
I was getting the same error as fade2grey (CPG 1.4.3, Win2K3, PHP 5, MySQL 5) and, at the advice of my webhost, I:
* Booted up phpMyAdmin
* Found the table "cpg143_categories"
* Set the DEFAULT column for "description" to NULL
... and then do the suggestions from Paver?
-
Did you try ...
... and then do the suggestions from Paver?
Hi. I already did the paver suggestions, but I had downloaded the 'fix' version of the script so I didn't think the 'null' change was necessary!
So, as a relative newbie to mysql and php, how do I access myphpadmin on my win2k system which is running php5 and mysql5?
Thanks.
-
You download it from phpmyadmin.net and then you install it.
-
I am silly. I was doing some searching and it was all there in front of me. I'm with you now! (sorry).
However, I'm running V1.4 and all my tables start with "coppermine"...
I presume I should be changing: copperminecategories?
-
If your database's name is "coppermine", then the tables will be in the form of "coppermine_categories". Note the underscore.
Easier to see once you get phpMYAdmin installed.
-
If your database's name is "coppermine", then the tables will be in the form of "coppermine_categories". Note the underscore.
Not quite correct. The database name, whatever it is, would go in front of the table name, separated with a period - if you wanted to explicitly state the database name. Usually you're in the database, so only the table name is necessary. The table prefix for Coppermine *does* go in front of the "categories" for the categories table. The prefix does not have to have an underscore, although most people do use the underscore. Judging from the error message listed, howie10's table prefix has no underscore.
Easier to see once you get phpMYAdmin installed.
Quite true.
-
Well. It worked! Thanks very much.
Only one thing, I had the structure wrong in the first place, so I deleted all the categories and tables in comppermine and started again. However, timer.php re-creates the cats and albums, - but then skips all the file uploads because it says they already exist!
I've even tried uploading a filesystem with different folder names, - and then overwriting it with my original one, but it still thinks the files exist.
Any ideas?
-
If you only deleted the categories & tables, you didn't delete the pictures, did you? You need to delete the categories & albums & pictures to start from scratch, then use timer.php to do its thing. Then in the future, you add files & folders and run timer.php to add only the new ones. Make sure you don't rename any categories or albums or timer.php will think the folder-named categories & albums don't exist and it will recreate them. That's my main plan for a plugin; to take care of renames in Coppermine.
Also make sure you don't add folders in such a way that a previous album becomes a sub-category, if that makes sense. You have to keep the levels straight since there are no sub-albums.
-
Ahh. Yes, it all makes sense, however, I now can't delete the pictures because I have a beautiful category structure - with no pictures in it. However, the main page footer tells me that there is 1600 pictures in 106 albums and 43 categories.
Somewhere, the application is holding info on the original pictures and thinks they are still there? (I suppose).
???
-
Well I assume you emptied the tables directly in phpMyAdmin? If so, you did something Coppermine cannot count on since it would never do that itself. In effect, you hacked the database. Empty the album table, the category table, and the pictures table, then try again.
-
You are correct. I did end up deleting categories using phpmyadmin. However, I did this because when I tried to do it in coppermine, they were locked and couldn't be deleted. Sorry, I can't remember the exact error message.
I have just deleted everything from the 3 databases as you suggest, and it's worked a charm. I appreciate your patience. You're a star!
This is an awesome mod. Thanks.
H.
-
Nibbler's the star for the mod; you're welcome for the support.
I find this mod extremely useful as well and hope to make it more robust some time. Others are welcome to do it instead or in addition - my toes are tough.
Oh, since some words were incorrectly used in the past few posts, here's a quick overview:
- database: a collection of tables
- table: a collection of records with fields
- record: a collection of values in fields (a.k.a. a "row" in spreadsheets)
- field: a "column" in spreadsheet parlance
-
That's really useful info. And thanks again to Nibbler! ;D
I am only having one slight problem now; timer stops after a few minutes (appears to crash). Not a hang, but the browser just goes to the 'no webpage found, - check dns setting etc), error.
I don't suppose there's a method of logging the script so that I can see which file/s causes this is there?
Logging would also be good to give a list of files it fails to load, so that I can manually convert formats etc. (just a suggestion).
Thanks again.
H.
-
Depending upon your webserver's PHP settings, this is probably a PHP timeout issue. There are other threads that discuss this. If you cannot adjust your PHP settings, then you have to work with what you have.
This script has a delay between requests ("sleep betwen additions") and a number of images to run on each page load ("limit additions per refresh"). Adjust these until things work and don't timeout. Safe values I would guess would be a not-so-long delay (500-1000 ms) and a small number for refresh (5 or 10), depending upon how long it takes to create each thumbnail & intermediate image.
Yes, it would be good to log the operation of this script, ideally to the regular Coppermine log. There are probably a few lines you could add to do that now, but I don't have time right now to check that.
-
Thanks again for the reply.
Well, I've adjusted my timeouts in php.ini to 600 and also the filesize to 8m.
I've also changed my apache conf file timeout from 120 to 600. - All to no avail.
I wonder if (as a temp workaround), I can log into coppermine and upload the files in the empty directories using the standard coppermine interface. Or will this confuse things the next time I run timer.php?
???
-
Weird. I don't know why you are having trouble.
Yes, you can use any of Coppermine's upload tools. Timer.php uses Coppermine's table to figure out if the picture already exists. As long as you use the same structure in both category & albums versus the folder structure, you can use either timer.php or Coppermine to add files and both will be happy. In fact, timer.php uses Coppermine's functions to add everything; it just gathers things up at the beginning to automate the process you would do manually.
-
Tis wierd isn't it. Even more so because I run timer.php with a completely different directory structure and it works fine!
Good that I can do a few manual uploads though. I'll pick out the largest files in each directory first.
Couple of Q's, - when you have time !! :-
1/ Is it possible to 'break' the timer script if I set an album to be read only by a specific group?
2/ Can it crash if a file exceeds the max width/heigh in cpg config?
3/ Are there any known filetypes which cause conflicts? The directory structure I am testing might have some unsupported filetypes, whether by GD2 or coppermine.
4/ Is there also a timeout setting in MySQL5 which might need tweaking?
TIA, - as always ;)
-
This is one reason we should use a separate board for this one add-on. I'll put together a pre-1.0 release plugin for this script and post it so we can start a sub-board on http://cpg-contrib.org/board/ (http://cpg-contrib.org/board/). (Donnoman mentioned this earlier in this thread.)
One thing to note is that timer.php does not check for problematic characters in your directories. Check for apostrophes or other "forbidden" characters (as noted in the coppermine config setting). If you can figure out why one directory structure works but the other doesn't, please post because that will probably be useful for developing this script further.
I don't have any answers to your questions off the top of my head. I'll check into it later when I have time, if no one else answers.
-
Thanks. I promise to post back here too.
8)
-
And here I am ;)
There is a problem i've discovered. Although the script creates albums under categories and sub categories in the correct way, the albums in coppermine are not classified under any category when you look at their properties.
As a result, normal http uploads are very difficult to assign to the correct album and if there are different albums with the same name in different directories, it is impossible to tell (from the album list in uploads, anyway), which are which.
H.
-
Hi.
I have discovered another problem:
After I have run timer.php, I cannot remove any of the albums it created. I have tried to remove an album in coppermine, - so that I don't confuse the databases), before I delete the matching folder in my directory tree.
However, when I try to remove the album in cpg, I get an error message telling me that I don't have write access to it. Although, If I delete any other albums which I made in the standard way, (in coppermine), I have no problems.
I am running Apache 2 with mysql5 on a win2k installation. My root folder (under 'albums'), is set as write permissions for everyone, in 'sharing'.
I would certainly appreciate any pointers ???
-
timer doesn't modify the file permissions. It's probably a result of how the images were originally placed in the albums directory.
make sure all folders have been chmodded to 777
-
I've created a plugin conversion of timer.php called CPGMassImport.
you can find more information about it on cpg-contrib.org
http://cpg-contrib.org/board/index.php?topic=123.0
The new version is downloadable from sourceforge, check the link above for more information.
It will also be available via CVS.
Please keep the discusions of CPGMassImport on the cpg-contrib forums so as not to make this thread any more confusing than it already is.
-
timer doesn't modify the file permissions. It's probably a result of how the images were originally placed in the albums directory.
make sure all folders have been chmodded to 777
This is a windows 2k machine (Apache2). I can't chmod anything. I've set all folders and subdirectories to be writeable. Weird.
-
I dont understand this Thread - and i read it completely.
All i want to do is :
Move all Pictures and Categories from MKPortal/modules/coppermine/albums to /gallerie/albums with the same Filenames etc.
I uploaded the timer.php to my /gallerie/ and started it and nothing happens.
Only Access denied
- no GUI - nothing.
Where is the Instruction for this Importer ?
Update :
Will use this one : http://cpg-contrib.org/board/index.php?topic=123.0
-
when i try and use it in a fresh install of 1.4.2 every photo fails..
-
don't use cpg1.4.2, it's outdated. Post a meaningful error message, link and more details. Just saying "doesn't work" is not helpful for supporters at all.
-
ok thanks! the only reason i was using it was because my fantastico had it as the basic install.. i've now upgraded..thank ya
-
so since this mod is for 1.4.3 i'm guessing it wont work with 1.4.5?? or is there an upgrade anywhere? i have over 2500 pictures to upload and i'm not looking forward to it.. this batch upload would be great!
-
This thread is marked as 1.4.x so it should be fine. I'm using timer.php fine with 1.4.5 (although I have modified it for my needs although not since 1.4.2).
However, you should consider using the new MassImport plugin which is based on timer.php. See donnoman's post above and the link: http://cpg-contrib.org/board/index.php?topic=123.0 (http://cpg-contrib.org/board/index.php?topic=123.0). (I am considering using it as well but I need to incorporate my mods to timer.php into the plugin so I'm waiting until I have time to do so.)
-
i was having a problem where out of no where my photos would fail to add.. but i just uninstalled the mod and reinstalled it and it stopped failing.. incase someone else has this problem.. hope that helps
-
Quote:
[Note: Tagged as 1.4, see later posts in thread for updated versions of this mod]
Why not put in in the first entry? Makes life simpler...
-
This mod has been made for cpg1.4.x, so it should work for any version (cpg1.4.0, cpg1.4.1, cpg1.4.2, cpg1.4.3, cpg1.4.4, cpg1.4., cpg1.4.5, cpg1.4.6, cpg1.4.7, cpg1.4.8, cpg1.4.9, cpg1.4.10), as suggested above. Since the cpg1.4.x series is the most recent stable one, there's no need to change anything imo. However, it's mandatory that you don't run outdated version, so your version should be cpg1.4.10. If you have an older version: upgrade. If this mod doesn't work with cpg1.4.10, please post details.
-
I just wanted to add that I got the script in the first post to work by adding "cpg_" before each call to db_query.
-
[Note: Tagged as 1.4, see later posts in thread for updated versions of this mod]
I took Flux's timer.php and merged a couple of my changes to his code. This is tested with cpg 1.3, for 1.4 you should just need to change the db_query's with cpg_db_query's.
-
Does anyone know how to modify this script so that it will create albums NOT in categories. I want to create a directory under albums and then have an album in no category created matching the folder name?
Any help would be greatly appreciated.