forum.coppermine-gallery.net

No Support => Modifications/Add-Ons/Hacks => Mods: Uploading => Topic started by: Nibbler on January 21, 2005, 11:37:57 am

Title: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on January 21, 2005, 11:37:57 am
Here we go, upload this into your coppermine directory and call it timer.php (I hardcoded it, so sue me :P) then go there, you know the drill.

It will keep reloading until all the pics are added if you set it to, so you can start it and leave it unattended for hours until it finishes, even with php set with a very low max_execution_time (ignore the fatal error). Not recommended for shared hosting environments unless you are vewy vewy careful. :)

Note: Auto reload needs js support.

[Note: Tagged as 1.4, see later posts in thread for updated versions of this mod]

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: mrplow on February 08, 2005, 10:13:33 pm
You, sir, are a legend.

 ;D

I thank you for your sterling work.




I can't express how awesomely happy I am with this hack!  :D :D :D


edit (heh..)
EXCEPT... it's a shame it can't resume a folder/cat... if I bung a load more files into the same folder then ask your amazing script to do its work again, it starts a new catagory and populates it with new albums, all of the same names as before, and starts from scratch.
Shame about that :(
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: thumeau on February 10, 2005, 05:09:38 pm
Thank you so much Nibbler,

So glad to be able to batch input more than 1 album at a time.

I can't get to have the autorun / auto reload working eventhough JS is definitely on.
I tried both on Mozilla Firefox and IE...
Any ideas?

Thanks,
Thomas
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: oshelef on March 08, 2005, 06:19:46 pm
I get the following error when I try to run this:


"Created root category

Warning: Invalid argument supplied for foreach() in /var/www/html/coppermine/timer.php on line 200
structure created, 0 files to add."

The category is created, but no albums or pictures are included. (I've gotten more complicated errors before, but I got the above trying to use it add a single image, in a single directory)

I.e. directiory albums/test contains test.jpg and nothing else. permissions are all 775.

Any suggestions as to what might cause this?

Thanks,

Orie

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on March 08, 2005, 06:34:49 pm
OK,  you make a folder in albums -> that becomes the root category.
You cannot then have any pics directly in that folder since you cannot have pics directly in a category.
You need to make folders inside the root folder - these become your albums, or more categories.

Bottom line is - You can only create a directory/file structure that would be a legal category/album/pic structure in coppermine.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: oshelef on March 08, 2005, 07:27:46 pm
Ahh, I think I understand it. I need a folder structure that looks like:

Category/Album/Image.jpg right?

I redid my test and it worked; although it still spit out an error, it did insert my test image as it should.

Edit, it works with one image, but when I have more than one image I get this following error:

"Fatal error: Call to undefined function: array_diff_assoc() in /var/www/html/coppermine/timer.php on line 98
structure created, 700 files to add."

Thanks for your help.

Orie
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on March 09, 2005, 11:48:02 am
Ah, seems the script requires PHP 4.3.0 or better to work. You can try adding the substitute function in the first comment here (http://uk.php.net/function.array_diff_assoc) into the script, or you could upgrade your PHP, or you could try using PHP_Compat (http://pear.php.net/package/PHP_Compat).
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on March 14, 2005, 07:05:34 am
Nibbler - Great script.

"It needed more power; so I rewired it"  (Tool Time Joke, hope somebody gets it)

I modified the script so that #1 you can run it against the album directory itself as the root cat.. There's an exclusion list that eliminates the basic garbage folders you might find there.

#2 It eliminates trying to add thumb_ and normal_ images that are already there from previous batch attempts.

#3 Checks to see if it can use a pre-existing cat/album before creating the new ones.

Which allows you to run this to do your batch adds all the time, successive runs don't hurt anything.

I was using this script against 1.4, but I didn't change anything that would be materially different than 1.3 so I would expect it to work on both platforms.

Also a nice little side effect of running this with 1.4 is that IPTC titles,keywords, and comments are automatically imported if you have iptc turned on in the config.

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on March 14, 2005, 07:12:25 am
@mrplow With my changes there should be no problem resuming. Do it as many times as you like.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juniors on March 23, 2005, 09:51:15 pm
Edit (double Post)
......
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juniors on March 23, 2005, 09:52:13 pm
Hello

With both Scripts i get the following Error when i call the timer.php from my Browser:

Quote
Parse error: parse error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING in /abc/def//timer.php on line 174

Any hints?

Thank you...
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on March 24, 2005, 04:51:29 am
im sure the line number that errors out isn't the same between the two scripts. make a note of which script gives which line number.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: kegobeer on March 24, 2005, 05:15:54 am
This is probably a problem with the file format.  I've attached a plain text version of Donnoman's file.  Try this one and see if you still get those errors.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juniors on March 24, 2005, 12:04:12 pm
Hi Kegobeer

Thank you for the Script. With this one i recieve the following Error:

Quote
Parse error: parse error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING in /abc/def/timer.php on line 416

Any hints again?

Junior
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on March 24, 2005, 02:13:38 pm
That file is 266 lines long. Therefore there cannot be an error on line 416. I suggest you try a different ftp client.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juniors on March 24, 2005, 03:59:02 pm
Hi

Iam sorry. I fixed the Script. Now its exactly 266 lines long and my Error-Msg is:

Quote
Parse error: parse error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING in /abc/def/timer.php on line 192

Line 192:
Code: [Select]
$auto = (isset($_POST['auto']) && $_POST['auto']) ? 'checked = "checked"' : '';
 ???
Code: [Select]
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: kegobeer on March 24, 2005, 06:12:43 pm
Are you uploading your files as ASCII or binary?  These files need to be uploaded as ASCII, or use the auto switch so your FTP client will do it for you.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juniors on March 25, 2005, 01:03:29 pm
I uploaded the filles in ASCII of corse  ;D
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: kegobeer on March 26, 2005, 02:25:53 am
Then perhaps you should try a different FTP client.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: flux on April 11, 2005, 03:28:55 pm
Hi,

here's my small participation to this nice and very usefull script..

Unfortunately as i didn't read the whole topic and especially donnoman's posts i didn't see that some of my modificatons had already been introduced and i just grabbed the file at the top of the topic, expecting it to be the latest version.. i was quite in a rush and started to make my modifications right away, silly me :)

Here's how the script works :
The script expects a directory name under ./albums/, that directory is supposed to be a root category in your gallery. Bottom directories, that should contain your pictures, will be treated as albums and intermediate directories will be treated as sub categories from the root category of course..
here's a quick example, let's say you have the following root category tree :
./albums/lib/
you should declare "lib" as the root directory to process.
now if you have the following tree under lib :
./albums/lib/A/Aname/pic.jpg

a root category named "lib" will be created, a sub category "lib>A", name "A" will be created, an album "lib>A>Aname" named "Aname" will be created and the picture pic.jpg will be added to the album "lib>A>Aname"..

that's how the original script works..

Now here are the features i've added to the original script :

In the same way donnoman did it, i added some filtering in order to avoid adding the same category/album/picture twice. So if you run the script twice on the aboce dirctory tree, everything should be created/added the first time and nothing will be done the second time..

Beware that there is a "limitation" in my database query on how category and album names are matched. Indeed a side efect of the query will be to limit creation of categories and albums to unique names. Which is not a limitation in the orignal design of CPG but is obviously a limitation of the filesystem (you cannot have the same directory name twice at the same location).. So you cannot have two categories or albums with the same name at the same location.. For example if you manually create, thru cpg interface, a second "Aname" album under "lib>A" and run the script to add new pics located in "/lib/A/Aname/" the script will use the first ID of the albums named "Aname".. maybe it's not worth mentioning, i don't know.. The same thing will happen with Category Names..
Of course if your second "Aname" album is located under "lib>Amisc>" then it's not a problem as the script will always use the reference to the parent when it tries to find if an album or category name already exists.. usually different parents mean different children ;)


Then the pictures filter was already there in the original script but was applied while processinf every pictures found.. I've add to that filter a global pictures filtering that will remove from the pictures lists build on the directory tree, the pictures already inserted in the database.. So for example if you update the directory tree under"./albums/lib", let's say it contains 20k+ pics but only have 150 new pictures, then my script will purge the already added 20k+ pics and add directly the 150 new pics.. quite nice isn'it  :)

A thing you shoul also know it that i've modified the way filenames are build.. In the original script any char not being part of the lists a-z A-Z 0-9 would be replaced by the char "_" (underscore).. I've improved that by removing existing double underscores "__" until there aren't any doubles left..

You'll probably notice that the renaming process will crush any non US chars, so you might want to modify that if you filenames have international chars in it..

Well i think it's all i can say about it..
I've tested it against a quite huge fake directory tree made of 26 subcategories (a-z) and 26x26 albums (26 albums in each 26 subcategories), each album containing 50 times the same picture named from 1.jpg to 50.jpg, which make a total of 33800 pictures..

The script take a few seconds (25sec) to build and filter the directory tree, which is located on a remote nfs server, then it will process the pictures according to the reloading parameters you've chosen.. The first update took a few hours as expected.. Then adding a few pics here and there just take a few seconds, no need to go over the previously added 30k pics right ? ;)

Thx for the coder of the original script..
Enjoy,

flux
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juanito on April 14, 2005, 06:11:05 am
i get the following error message:

Warning: array_diff_assoc(): Argument #2 is not an array in C:\WebRoot\photogallery\timer.php on line 102
After filtering 0 pictures will be added
structure created, 0 files to add.

im trying to use flux's versio... plz help!!  :(

the original one by nibbler works great though but the other 2 modified versions are giving me errors. it'd be nice to have the extra features. any help is appreciated. thanks!
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: moulari on April 17, 2005, 05:18:00 pm
The idea for the timer script is what i have been searchin now for almost a year for my CPG, unfortunatly none of the three works for me.
Since I was feed up with the normal upload process my gallery stopped at 615 Pics -> although on my local disc i got 3000+. So now i did as discribed set up a directory structure under albums/test/test/ and uploaded some random pics for a quick test - but all the three tested scripts give me the following back "Using root Category
structure created, 0 files to add."  -> But there is no structure created.

So schould i use a fresh install of CPG for the scripts or are there any special options in CPG which i schouldn't use?

Thx for the script anyway hope it works for me too soon

Setup:
Apache 1.xxx
PHP 4.3.8
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on April 17, 2005, 05:22:40 pm
Make sure you setup a legal file structure - check the previous postingt about this.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: moulari on April 18, 2005, 11:06:01 am
thx for the tip

my directory structure looks like this (nevermind the word since its german)

/albums/Schloss/Sommer/xxxxxx.jpg

So is this legal or do i have to change something?

Solved it: installed cpg1.4.1 now the script isrunning flawless
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: moulari on April 20, 2005, 11:45:03 pm
Ok maybe my flawless was to fast written  ;D

After a night of running, i counted 2549 files in 53 albums and 28 categories added. When comparing the structured in CPG with the directory structure in /albums. I found one directory which wasn't added, i'm not speaking of userpics and edit, I mean a drectory with a legal structure.

the structure would be: /albums/Urlaube & Reisen/Wagrain Skifahren/Februar03

when running again found out the, creation process for albums and categorys stops and displays a warning message and stops. Pls check attached picture for the look of the message.

any suggestions??
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Tranz on April 21, 2005, 03:41:29 am
The '&' in the folder name may be causing the issue. Try using something else.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: moulari on April 22, 2005, 12:23:24 am
The & can't be the error because it is in another categoryname too and that one was addede correctly.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on April 22, 2005, 02:30:44 pm
So enable debug mode and post the actual mysql error message ;)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: moulari on April 24, 2005, 11:46:17 am
thx for the debug tip, solved the problem. The error was in another directory with an album name with an invalid character.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: tramfam on April 25, 2005, 03:51:46 am
Thanks to all who've done a great job on this script. I started with a fresh install of Coppermine and the scripts would create the categories, but not populate the albums. I realized that the scripts fail if there are no pictures already defined in the database!

So import a single file by hand to ensure that at least one picture is in the gallery. Then proceed with the timer script!

Thanks,
Mark
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: londonhogfan on April 29, 2005, 08:37:46 pm
works great.  Installed from scratch and have uploaded over 1,000 pics with no problems.  Saved me soo much gime.

www.razorbacklegacy.com/hog_archive (http://www.razorbacklegacy.com/hog_archive)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: niuck on July 01, 2005, 07:37:49 pm
I'm having problems with the original script. It doesn't upload the pictures, just creates category and albums.
My dirs look like this: /var/www/localhost/htdocs/fester/albums/Categories/Albums/Pics.jpg <- thats okey right? It's all chmodded to 777.

This is what i get when i click begin in the script:.
Code: [Select]
Created root category
created album 000429_Valborg
created album 000517_Boras
created album 001012_Forfest_Kaneback
created album 001105_Kanarieoarna
created album 000101_Fest_Atvid
created album 000101_Pokerkvall_David
created album 000205_Pokerkvall_Bigalow

Notice: Undefined index: auto in /var/www/localhost/htdocs/fester/timer.php on line 175
structure created, 325 files to add.
When i click continue to convert the pictures nothing happens.

Here is some info:
Code: [Select]
USER:
------------------
Array
(
    [ID] => 826d834b7f0764b5223187fec79ce0d4
    [am] => 1
    [liv] => Array
        (
            [0] => 236
            [1] => 176
            [2] => 229
            [3] => 223
            [4] => 209
        )

)

==========================
USER DATA:
------------------
Array
(
    [user_id] => 1
    [user_group] => 1
    [user_active] => YES
    [user_name] => niuck
    [user_password] => ********
    [user_lastvisit] => 2005-07-01 18:57:34
    [user_regdate] => 2005-06-29 05:51:31
    [user_group_list] =>
    [user_email] =>
    [user_website] =>
    [user_location] =>
    [user_interests] =>
    [user_occupation] =>
    [user_actkey] =>
    [disk_max] => 0
    [disk_min] => 0
    [can_rate_pictures] => 1
    [can_send_ecards] => 1
    [ufc_max] => 3
    [ufc_min] => 3
    [custom_user_upload] => 0
    [num_file_upload] => 5
    [num_URI_upload] => 3
    [can_post_comments] => 1
    [can_upload_pictures] => 1
    [can_create_albums] => 1
    [has_admin_access] => 1
    [pub_upl_need_approval] => 0
    [priv_upl_need_approval] => 0
    [group_name] => Administrators
    [upload_form_config] => 3
    [group_quota] => 0
    [can_see_all_albums] => 1
    [group_id] => 1
    [groups] => Array
        (
            [1] => 1
        )

)

==========================
Queries:
------------------
Array
(
    [0] => SELECT extension, mime, content FROM cpg133_filetypes;
    [1] => SELECT * FROM cpg133_users WHERE user_id='1'AND user_active = 'YES' AND user_password != '' AND BINARY MD5(user_password) = '48f8d008981572fd9fc3218c2460c833'
    [2] => SELECT MAX(group_quota) as disk_max, MIN(group_quota) as disk_min, MAX(can_rate_pictures) as can_rate_pictures, MAX(can_send_ecards) as can_send_ecards, MAX(upload_form_config) as ufc_max, MIN(upload_form_config) as ufc_min, MAX(custom_user_upload) as custom_user_upload, MAX(num_file_upload) as num_file_upload, MAX(num_URI_upload) as num_URI_upload, MAX(can_post_comments) as can_post_comments, MAX(can_upload_pictures) as can_upload_pictures, MAX(can_create_albums) as can_create_albums, MAX(has_admin_access) as has_admin_access, MIN(pub_upl_need_approval) as pub_upl_need_approval, MIN( priv_upl_need_approval) as  priv_upl_need_approval FROM cpg133_usergroups WHERE group_id in (1)
    [3] => SELECT group_name FROM  cpg133_usergroups WHERE group_id= 1
    [4] => DELETE FROM cpg133_banned WHERE expiry < '2005-07-01 19:32:34'
    [5] => SELECT * FROM cpg133_banned WHERE ip_addr='192.168.1.1' OR ip_addr='192.168.1.1' OR user_id=1
    [6] => SELECT parent FROM cpg133_categories WHERE cid = '33' LIMIT 1
    [7] => UPDATE cpg133_categories SET parent='0' WHERE parent = '33'
    [8] => UPDATE cpg133_albums SET category='0' WHERE category = '33'
    [9] => DELETE FROM cpg133_categories WHERE cid='33' LIMIT 1
    [10] => SELECT cid FROM cpg133_categories WHERE 1
    [11] => UPDATE cpg133_categories SET parent = '0' WHERE parent=cid OR parent NOT IN (1)
    [12] => SELECT cid, name, description FROM cpg133_categories WHERE parent = '0' ORDER BY pos
    [13] => SELECT cid, name, description FROM cpg133_categories WHERE parent = '1' ORDER BY pos
    [14] => UPDATE cpg133_categories SET pos='0' WHERE cid = '1' LIMIT 1
    [15] => SELECT pid, filepath, filename, url_prefix FROM cpg133_pictures,cpg133_albums WHERE cpg133_pictures.aid = cpg133_albums.aid AND cpg133_albums.category = '33' AND approved='YES' ORDER BY filename
)

==========================
GET :
------------------
Array
(
    [op] => deletecat
    [cid] => 33
)

==========================
POST :
------------------
Array
(
)

==========================
VERSION INFO :
------------------
PHP version: 4.3.11 - OK
------------------
mySQL version: 4.0.24
------------------
Coppermine version: 1.3.3
==========================
Module: gd
------------------
GD Support enabled
GD Version bundled (2.0.28 compatible)
GIF Read Support enabled
GIF Create Support enabled
JPG Support enabled
PNG Support enabled
WBMP Support enabled
XBM Support enabled
==========================
Module: mysql
------------------
Active Persistent Links 0
Active Links 1
Client API version 4.0.24
MYSQL_MODULE_TYPE external
MYSQL_SOCKET /var/run/mysqld/mysqld.sock
MYSQL_INCLUDE -I/usr/include/mysql
MYSQL_LIBS -L/usr/lib -lmysqlclient 
==========================
Module: zlib
------------------
ZLib Support enabled
Compiled Version 1.2.2
Linked Version 1.2.2
==========================
Server restrictions (safe mode)?
------------------
Directive | Local Value | Master Value
safe_mode | Off | Off
safe_mode_exec_dir | no value | no value
safe_mode_gid | Off | Off
safe_mode_include_dir | no value | no value
safe_mode_exec_dir | no value | no value
sql.safe_mode | Off | Off
disable_functions | no value | no value
file_uploads | On | On
include_path | .:/usr/lib/php | .:/usr/lib/php
open_basedir | no value | no value
==========================
email
------------------
Directive | Local Value | Master Value
sendmail_from | no value | no value
sendmail_path | /usr/sbin/sendmail -t -i  | /usr/sbin/sendmail -t -i
SMTP | localhost | localhost
smtp_port | 25 | 25
==========================
Size and Time
------------------
Directive | Local Value | Master Value
max_execution_time | 999999 | 999999
max_input_time | 300 | 300
upload_max_filesize | 2M | 2M
post_max_size | 16M | 16M
==========================
Page generated in 0.237 seconds - 16 queries in 0.012 seconds - Album set :

Anyone knows what could be wrong?
I'm using 1.3.3 in Gentoo if that matters.

Thanks!
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Woofy on September 02, 2005, 11:26:57 am
Hi there !

I'm having some troubles with timer.php.
Well, I'm using Coppermine 1.3.3 , Apache 2, PHP 5 on a FC4 distro.

When I try to use timer.php, I get this Fatal error :

"Fatal error: Call to undefined function cpg_db_query() in /var/www/html/cpg/timer.php on line 241"

My "albums" directory's structured as following :

albums
   FTP
      categ
         album
            .jpg files

After a litlle search, I don't find any cpg_db_query declaration ... Is this hack working with cpg 1.3.3 ? Am I missing something ?

Thanks for your help !
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on September 02, 2005, 12:15:45 pm
Looks like you are using a version designed for cpg1.4. Which version of the mod are you using ?
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Woofy on September 07, 2005, 04:00:25 pm
I've tried the one post by Flux, and the mod posted by donnoman (your mod modified to enable retries).

In some posts, I've seen some answer for similar problems on several mods (between 1.4 and 1.3.x), and the solution was to replace cpg_db_xxx function call by db_xxx. Is that a solution for me too ?

!!! EDIT :

I've solved my problem : I've changed changing cpg_db_query function call to db_query which is the correct function nam in cpg 1.3.3.
My mod was effectively for cpg1.4, sorry for my mistake Nibbler (and for my poor english of french guy)  ;D
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on September 30, 2005, 06:26:53 am
I took Flux's timer.php and merged a couple of my changes to his code.  This is tested with cpg 1.3, for 1.4 you should just need to change the db_query's with cpg_db_query's.

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: flux on October 02, 2005, 12:27:00 am
lo,

@donnoman
what kind of changes did you add to my script exactly ?


i was stupid enough not to see that a more advanced script than the original one had been already made when i started mine..
basically the only thing i made in my script that may not have been in the previous script was to skip all the already existing pictures in the galleries at once, that increased the processing speed greatly.. besides that my script just made sure categories/albums didn't already exist before creating new ones, as the other script does..

And yes i made my script for 1.3 only so it shouldn't work with 1.4 and to make it work you only have to replace the db function name..
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on October 10, 2005, 09:37:12 pm
I liked your method of filtering out stuff that already had been imported. So I dropped most of my code in favor of yours. I just added the pieces that allow you to pass the intial directory as null and it will import everything under the "albums" directory.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: CLG on October 18, 2005, 01:23:07 pm
Hi,
I have some problem by adding with the postet scripts (i tryed all, from flux&donnoman). I must add 38 Gigabyte Pictures (50k+) and some pictures have a size of >5mb, all pictures under 1mb add the script without problems, when a bigger picture in the directory the whole script stop..

i have changes the php.ini of 2400 and 500mb, and have in the config from coppermine 6000kb, and the script can´t import a file with 1500kb, why this?

please help me.. pascal
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: flux on October 19, 2005, 02:06:22 pm
don't think it has anything to do with the scripts..

as far as i can tell you've covered with your settings the various sources of limitations (cpg & php)..
Now if you don't get any debug info it'll be hard to debug, i guess you should turn on dubug in cpg config and maybe you'll get some interesting output when the script fails..

But personnaly i'd put my money on the image component, not sure GD2 or image magic are bug free..

Once you've found some debug info you should ask some support in a support forum..

A good test would also be to try it thru cpg integrated batch add tool and see what happens, if it fails too then it has nothing to do with the script itself but more with a cpg core funtion maybe or with a server component (gd, php, apache and so on..) and don't forget to turn debug on in your cpg config..
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: andrez1 on October 20, 2005, 06:31:36 pm
I just added the pieces that allow you to pass the intial directory as null and it will import everything under the "albums" directory.

Hello.

Nice script. 

It works well, imports everything under /albums.  But doing so with a null value to 'SubDirectory (under albums)' results in
'filepath' in 'cpg***_pictures' starting with a slash.

If the picture is already there, by other methods: it will be inserted twice.

It produce adresses like http://**/cpg133/albums//***/**/*.jpg , double slash after albums.

Any way of filtering out things like '../../../..'   as possible value   'SubDirectory (under albums)' ?





Mvh Andrez1

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: flux on October 21, 2005, 12:43:41 am
i haven't analyzed how the directory structure is created but a quick and dirty way to solve the problem would be to simple remove a possible "/" first char in the path string just before insterting the picture..

i guess that having an openning "/" in the filepath data is not correct so the following simple patch would be okay.. although the proper thing to do would be to sanitize the file path when building the directory structure..

So using the latest Donoman file, below line 178 :
Code: [Select]
$dir_name = dirname($pic_file) . "/";
insert :
Code: [Select]
$dir_name = ( substr($dir_name,0,1) == "/" ) ? substr($dir_name,1) : $dir_name;
that should do it. If so please confirm it works fine, i have nothing to test it on here, the code works don't worry, just make sure it also works when using a subcategory info in the field..
then i'll post the updated file.

thx for your bug report

.flux
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: andrez1 on October 22, 2005, 12:51:44 am

So using the latest Donoman file, below line 178 :
Code: [Select]
$dir_name = dirname($pic_file) . "/";
insert :
Code: [Select]
$dir_name = ( substr($dir_name,0,1) == "/" ) ? substr($dir_name,1) : $dir_name;
that should do it. If so please confirm it works fine, i have nothing to test it on here, the code works don't worry, just make sure it also works when using a subcategory info in the field..
then i'll post the updated file.


That makes the script take a blank value.

It also works when  giving as a value a subdir containing another dir with imagefiles.

although the proper thing to do would be to sanitize the file path when building the directory structure..

Absolutely..

Tested a litle bit. 

the file (win):

c:\XXXXXX\htdocs\cpg133\albums\C370-3\TJA14\P8190003.JPG

in subdir C370-3\TJA14 is accepted at the  following values:

(aid,filepath,filename)      (given value to the script)

6236,\\c370-3\\/tja14/,P8190003.JPG        \c370-3\
6233,c370-3\\/tja14/,P8190003.JPG          c370-3\
6230,c370-3//tja14/,P8190003.JPG           c370-3/
6227,c370-3/tja14/,P8190003.JPG            c370-3
6224,\\c370-3//tja14/,P8190003.JPG        \c370-3/


The values given to the script results in 5 categories like the values given to the script. And a 'tja14' album in each.

The  'c370-3' value seems to be what the script is expecting. But it accepts  a lot (including '..', which might be a risk).



Mvh Andrez1
 
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: andrez1 on October 22, 2005, 01:44:08 pm


That makes the script take a blank value.



But filtering on what is already in the database is broken.. That does not work with the newly added line to the script,  and a null value passed to the scripts 'SubDirectory (under albums)'.

Filtering does work when the script is given a subdir as value.


Mvh Andrez1
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: flux on October 23, 2005, 03:14:09 pm
okay,

based on your comments i've made a few modifications to the latest version of donnoman's script.

it will now handle blank "subcategory" field properly and store valid paths in the database, without starting "/".

Filtering also works with blank "subcategory" field.

Concerning your previously insterted pictures using a blank "subcategory" field they were all stored with a faulty filepath so you'll have to fix them. The following sql query will do that for you :

Code: [Select]
UPDATE prefix_pictures SET filepath=SUBSTRING(filepath,2) WHERE RIGHT(filepath,1)='/'of course replace "prefix_pictures" table name by your own table name using your "prefix_" name..

that's about it..

oh yeah i've also added some more version checking in the script so it'll fail to start if you're not using cpg v1.3.x or 1.4.x
And it'll also work now for both versions 1.3.x and 1.4.x, i've just wrapped the db_query function BUT did some tests on a cpg1.3.5 version only !

So if you can confirm it works fine on a 1.4.1 beta too, i'll thank you for that ;)

.flux
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: flux on October 23, 2005, 03:16:35 pm
one last quick word, all my tests are done on a linux plateform so if there are some paths problems when running this script on windows, i'm afraid i won't be able to do much about it.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: andrez1 on October 24, 2005, 09:37:49 pm
it will now handle blank "subcategory" field properly and store valid paths in the database, without starting "/".

Filtering also works with blank "subcategory" field.


Works fine in cpg 1.33 an 1.41.

The 1.33 also gets filename as title.

The 1.41 do not getfilname as title. (and thats ok, can be set in config on thumbs)

Have not used the script for anything serious. Yet.  Nothing to repair.

As of now basic functionality is there, working. Fine to insert large amounts of pictures.

I have found 3000 pictures in a directory, presized and with thumbs_  present, to be the practical limit with 'batch add' on a local pentium4 - 3ghz -xp.  (and that is with standard values on script time in php).  More than that and 'reload' or new start on given directory is needed.

The 'batch add' gives possibility to set which album to insert pictures into.  This script does not. But it solves a problem with inserting large amounts of pictures. That is the use i consider. And when it's done something like -

UPDATE `cpg132_pictures` SET `aid` = 3 WHERE `filename` between 'xxx005.jpg' and 'xxx010.jpg'

(or whatever the WHERE clause and 'aid' is set to) - will  arrange the pictures in another order than the directory-structure.
(Hiding the albums originating from the script is part of idea.) 

The next time the script is run on the given directory it does not find the picture in db and do not filter it out. It tries to add it and reports something like 'picture already there'.  Thats OK, but not optimal.


Mvh Andrez1


Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: andrez1 on October 24, 2005, 10:14:25 pm
one last quick word, all my tests are done on a linux plateform so if there are some paths problems when running this script on windows, i'm afraid i won't be able to do much about it.

Not a windows-linux path problem. '..'  is understood as 'one directory up' in windows, linux and html.

'../themes' as a value to the scripts 'SubDirectory (under albums)'  gives a result i think not shold be allowed.

'/dir' or '/dir/'  or 'dir/subdir'  (given existing dirs) produces categories and albums and insert files into them. I think it should not.


I have a category 'c370-7/7' with the album 'tja22'  an a file like 'P8250003.JPG'

http://XXXX/cpg133/albums/c370-7/7/tja22/P8250003.JPG

That is produced by passing 'c370-7/7' to the script.  (dir and subdirs exist, there is nothing in the script to guide me,  stop me or warn me.)

I also have a  'c370-7' category with a '7' subcategory and 'tja22' album. Thats empty.  Should not be.


And so on. Gets messy.



Mvh Andrez1






Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: flux on October 29, 2005, 10:13:46 pm
not sure i understand why your previously added pictures are not filtered but then skipped with an "already added" output.. shouldn't happen, something must be broken with the filtering, although what i've tested here was working fine as far as i could tell..


as for "../" it's not supported and not filtered either because it's simply a bad use of the subdirectory field..
The script and the whole cpg system only deals with pictures placed in your "album" (aka fullpath in the config) directory directly under the root directory..

I guess it must have been stated somewhere in the first explanations of this batch add tool that the subdirectory field was made to target a directory under the "album" directory..
thus using "../" entries makes no sence as it would mean leaving the "album" directory tree..

that's all i can say..
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: andrez1 on October 31, 2005, 02:48:20 am
not sure i understand why your previously added pictures are not filtered but then skipped with an "already added" output.. shouldn't happen, something must be broken with the filtering, although what i've tested here was working fine as far as i could tell..

That happens on a cpg133 when pictures are inserted using the script and a value to 'SubDirectory (under albums)' (f.ex 'c370-8')

The dir 'c370-8' exist and have a subdir 'tja23' which contains pictures.

'http://xxxxx/cpg133/albums/c370-8/TJA23/normal_P8280001.JPG' is visible in coppermine after use of the script. The image is found in the category 'c370-8', album 'tja23'.  It is as it should be.

But there is no need to stick to the somewhat rigid order imposed by the script. 

The mentioned album 'tja23' have an 'aid' like '11250' , and a new album created like 'fancyname' with an 'aid' like '11251'.

Running  ---  UPDATE `cpg133_pictures` SET `aid` = 11251 WHERE `aid` like '11250'  --- on the db empties the album created by the script and move the pictures to another album. The album 'fancyname'. That kind of use is neccesary if someone want another name on an album other than the subdir the file resides in. That kind of use make the script useable.

The only thing changed is the 'aid' field in `cpg133_pictures`. 

the pair of 'filepath' and 'filename' is the same.


as for "../" it's not supported and not filtered either because it's simply a bad use of the subdirectory field..
The script and the whole cpg system only deals with pictures placed in your "album" (aka fullpath in the config) directory directly under the root directory..

I guess it must have been stated somewhere in the first explanations of this batch add tool that the subdirectory field was made to target a directory under the "album" directory..
thus using "../" entries makes no sence as it would mean leaving the "album" directory tree..

It can be a feature  ;) to be allowed to list the whole directorystructure of the disk (filesystem?) on a server running coppermine.

'../../..' to the script gives me a category '../../..' with a subcategory 'htdocs'.


'../../coppermine/albums/test1' to the script gives me a category '../../coppermine/albums/test1'  containing albums. From another coppermine-install that is..

If I ask for image properties on:

'http://xxxxx/cpg133/displayimage.php?album=7182&pos=0'

I get

'http://xxxxx/coppermine/albums/test1/alt/xxx001.jpg'

'filepath' on that picture is set to '../../coppermine/albums/test1/alt/'

A nice feature which can be exploited.

Mvh Andrez1
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on November 28, 2005, 06:07:25 pm
Should the image next to the thread say "1.4.x" as well?  Maybe the most current version should be added to the first message (with Nibbler's permission & an edit note of course)?

The current version is here:
http://forum.coppermine-gallery.net/index.php?topic=13949.msg105809#msg105809

I have some mods I'd like to suggest, like ignoring the "Thumbs.db" files Windows creates, a test run that doesn't create albums (to check things like when you rename an album in CPG and then timer.php wants to add a new album with the current folder name but now old album name - a mod I'm trying to decide how to address), and a mod to remove forbidden characters in directory names (like CPG does with filenames).  I'll add them here when I get the chance.  I'd also like to try to convert this into a plugin.  Has anyone attempted this yet?  Of course it needs to be more bulletproof and have more checks installed.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on November 29, 2005, 03:32:24 am
I'd be happy to host this project over on cpg-contrib.org, and we could have it made as a plugin, and available from cvs.

Perhaps a better name than TIMER might be appropriate ;)

When someone figures out a good name, I'll create the project.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juanito on December 09, 2005, 04:33:30 am
if its not too much to ask, would it be possible to get this nicely made timer script to include the mini thumbs and orgi back-up pix? im using Stramm's mod pack found here, http://forum.coppermine-gallery.net/index.php?topic=24341.0 and his mods add two additional files for each pix... one with the prefix mini_ and another with orig_ . And when I run the most recent timer script by flux, it detects these additional files as a completely new pictures and starts creating normal_ and thumb_ files for them. So would it be possible to get the script to include or perhaps simply just ignore the orig_ and mini_ files? If it involves alot of scripting, then no worries.. ill just stop using mini thumbs and watermarking. thanks!
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on December 09, 2005, 04:50:54 am
So would it be possible to get the script to include or perhaps simply just ignore the orig_ and mini_ files?
I haven't used Stramm's mod pack, but I did modify the timer script to ignore the "Thumbs.db" files that Windows creates when you browse Thumbnails in a directory.  The script above already ignores the thumb_ and normal_ files that Coppermine creates, so you merely have to add orig_ and mini_ to the 2 if statements that do this.

In the createstructure function, look for lines similar to these and modify as shown:
Code: [Select]
foreach ($contents as $file) {
if ( (strncmp($file,$CONFIG['thumb_pfx'],strlen($CONFIG['thumb_pfx'])) != 0 ) && ( strncmp($file,$CONFIG['normal_pfx'],strlen($CONFIG['normal_pfx'])) != 0) && ( strncmp($file,'orig_',strlen('orig_')) != 0) && (strncmp($file,'mini_',strlen('mini_')) != 0)) {
$filelist["$path/$name/$file"] = $aid;
}
}

Then in the addpic function, modify the if statement similarly:
Code: [Select]
if ( (strncmp($file_name,$CONFIG['thumb_pfx'],strlen($CONFIG['thumb_pfx'])) != 0 ) && ( strncmp($file_name,$CONFIG['normal_pfx'],strlen($CONFIG['normal_pfx'])) != 0) && ( strncmp($file_name,'orig_',strlen('orig_')) != 0) && (strncmp($file_name,'mini_',strlen('mini_')) != 0)) {
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Juanito on January 05, 2006, 06:37:09 pm
Thanks! Works like a charm.  :)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: fade2grey on January 09, 2006, 11:54:56 am
Hi guys, I'm have some problems with this - probably user error...

so far I've grabbed the file attached to the first post in this thread (updated Jan 4th 06)
copied it to my coppermine folder as timer.php
when I run it in a browser, I get a load of garbled PHP & a few boxes/fields to complete - nedless to say, it does nothing.
I've uploaded some test files in the format:

FTP_Upload
    Test
         10_01_2005
         11_01_2005
etc
the intention is for them to have the nesc catagory created & them to be posted in the Test Album.

Any suggestions or quick pointers?

A
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on January 10, 2006, 12:47:57 am
instead of grabbing the first post, why dont you go through the thread and grab the LAST updated copy, as most of the bugs and 1.3 and 1.4 compatability was worked out in the later versions.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: fade2grey on January 12, 2006, 02:57:56 pm
Okay, I've grabbed the most recent one from this thread & I get the following when Debug is turned on:

While executing query "INSERT INTO cpg143_categories (pos, parent, name) VALUES ('10000', '0', 'Test')" on 0

mySQL error: Field 'description' doesn't have a default value

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on January 15, 2006, 09:08:36 pm
sounds like your on Mysql 5, search the board, it's been discussed before.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Prisoner_24601 on January 19, 2006, 03:34:52 am
I was getting the same error as fade2grey (CPG 1.4.3, Win2K3, PHP 5, MySQL 5) and, at the advice of my webhost, I:

* Booted up phpMyAdmin
* Found the table "cpg143_categories"
* Set the DEFAULT column for "description" to NULL.

Ran the newest version of timer, again. 

Now, PROVIDED:

* I had a folder under "albums" called "testy" and
* A folder under "testy" called "testy2" (so the directory structure looks like: <root>\cpg143\albums\testy\testy2")

The TIMER script would create the category "testy" with an album in it called "testy2".

That's the good.  The bad is it wouldn't upload the pictures in "testy2".

When I turned the debug on, I'd get the following error:

Code: [Select]
While executing query "INSERT INTO cpg143_pictures (aid, filepath, filename, filesize, total_filesize, pwidth, pheight, ctime, owner_id, owner_name, title, caption, keywords, approved, user1, user2, user3, user4, pic_raw_ip, pic_hdr_ip, position) VALUES ('12', 'testy/testy2/', 'a.jpg', '14702', '17383', '362', '295', '1137635330', '1', '[my server name, edited out]','', '', '', 'YES', '', '', '', '', '', '', 'a.jpg')" on 0

mySQL error: Out of range value adjusted for column 'position' at row 1

File: D:\Inetpub\wwwroot\[my server]\cpg143\include\functions.inc.php - Line: 248

============

Notices:

\include\themes.inc.php

    * Warning line 1167: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)
    * Warning line 1168: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)

\include\functions.inc.php

    * Warning line 162: Cannot modify header information - headers already sent by (output started at D:\Inetpub\wwwroot\[my server]\cpg143\timer.php:10)

Little file named "a.jpg".  All of 12K.

I tried editing out the "pos" statements in timer.php and then deleting them altogether.  I tried changing the DEFAULT column to null and not null or other numbers in the cpg143_pictures table with phpMyAdmin and nothing helped.  I tried combinations of the above, too.

Any suggestions?  (My webhost does not wish to downgrade MySQL to 4.x.)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on January 19, 2006, 04:31:44 am
Something's weird.  The error is being caused because the field position is a number field and cannot be assigned to a string like 'a.jpg'.  I checked the timer.php script and it has an error in that it sends the filename where the position should be because it has an extra parameter in the function call.  But it works fine for me.  Maybe MySQL 4 will realize you are sending a string and ignores it whereas MySQL 5 won't let you do this and gives the error.  Or maybe both servers have a strict setting and mine has it off whereas yours has it on.

In any case, look for this line in timer.php:
Code: [Select]
if (add_picture($aid, $dir_name, $sane_name, $file_name)) {and replace it with this:
Code: [Select]
if (add_picture($aid, $dir_name, $sane_name)) {
Let us know if it works.  Since I don't have this issue, I cannot easily test it.  I assume the error in timer.php is due to an earlier prototype for the add_picture function.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Prisoner_24601 on January 19, 2006, 04:54:04 pm
Paver deserves WAY more karma points.

That worked!  Once again, using the directory structure I mentioned above, it created the album and category, but this time it uploaded my file and added it to the CPG database.  No errors.

I do want to test further, because the script didn't (earlier, before Paver's changes) skip already added files/galleries, as "was advertised" -- and I want to test with lots of JPEG's -- but I think this is probably gonna work pretty darn well.

I'll keep y'all updated on the progress.

(Also remember, if you're trying this on CPG143, that you have to change the lines that say "db_query" to "cpg143_db_query".  That's on pg. 2 of this thread.)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on January 19, 2006, 05:10:12 pm
(Also remember, if you're trying this on CPG143, that you have to change the lines that say "db_query" to "cpg143_db_query".  That's on pg. 2 of this thread.)
Actually, it's cpg_db_query.  But the best thing to do is to download the script that's compatible with both 1.3 & 1.4 that's somewhere in this thread.  The flux_donnoman mod timer script.  I don't feel like looking for it now.  That's the one I've been using.

I added a "check files only" option so that I can do a trial run that won't add anything.  I always use that before running a real run of the timer script.  That way I'm sure I didn't add an extra folder here or there or have a filename with quotes etc.  As I mentioned earlier in this thread, I'd like to contribute my additions but I think it's time to make this into a plugin and I haven't quite done that yet - that's when I plan to submit them.  Others are welcome to steal my thunder, or we'll see when I get to it.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Prisoner_24601 on January 20, 2006, 06:37:51 am
I went back and got the Flux-Donnoman one instead of the one in the first post in this thread (which says it's the newest) and updated it as you describe.  Works just great -- and I tried a couple hundred pictures and categories.

The only thing I did have to do was ask my webhost to increase the timeout in PHP.INI.

Now I'm off to search if there's a plugin that improves the response of catmgr.php ....  ;D
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on January 31, 2006, 09:58:31 pm
Hi. I love the idea of this mod.  :P

Has anyone got it working with a local windows installation, (win2k)?

I am logged on through http://localhost/, and when I do http://localhost/coppermine-install-directory/timer.php I just get the standard 'error while processing..' error which (in debug mode) gives an initial message of:

  While executing query "INSERT INTO copperminecategories (pos, parent, name) VALUES ('10000', '0', 'personal')" on 0
  mySQL error: Field 'description' doesn't have a default value

I do run MYSQL5, but I have followed the instructions above already in the hope that it solves it. Also, I have set the folder I made under /albums as writeable by everyone.

Thanks (in advance).
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Prisoner_24601 on January 31, 2006, 10:06:59 pm
Did you try ...

I was getting the same error as fade2grey (CPG 1.4.3, Win2K3, PHP 5, MySQL 5) and, at the advice of my webhost, I:

* Booted up phpMyAdmin
* Found the table "cpg143_categories"
* Set the DEFAULT column for "description" to NULL

... and then do the suggestions from Paver?
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on January 31, 2006, 10:28:41 pm
Did you try ...

... and then do the suggestions from Paver?

Hi. I already did the paver suggestions, but I had downloaded the 'fix' version of the script so I didn't think the 'null' change was necessary!

So, as a relative newbie to mysql and php, how do I access myphpadmin on my win2k system which is running php5 and mysql5?

Thanks.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Nibbler on January 31, 2006, 10:38:51 pm
You download it from phpmyadmin.net and then you install it.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on January 31, 2006, 11:07:48 pm
I am silly. I was doing some searching and it was all there in front of me. I'm with you now! (sorry).

However, I'm running V1.4 and all my tables start with "coppermine"...

I presume I should be changing: copperminecategories?
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Prisoner_24601 on January 31, 2006, 11:51:27 pm
If your database's name is "coppermine", then the tables will be in the form of "coppermine_categories".  Note the underscore.

Easier to see once you get phpMYAdmin installed.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on January 31, 2006, 11:57:12 pm
If your database's name is "coppermine", then the tables will be in the form of "coppermine_categories".  Note the underscore.
Not quite correct.  The database name, whatever it is, would go in front of the table name, separated with a period - if you wanted to explicitly state the database name.  Usually you're in the database, so only the table name is necessary.  The table prefix for Coppermine *does* go in front of the "categories" for the categories table.  The prefix does not have to have an underscore, although most people do use the underscore.  Judging from the error message listed, howie10's table prefix has no underscore.

Quote
Easier to see once you get phpMYAdmin installed.
Quite true.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 01, 2006, 02:59:11 am
Well. It worked! Thanks very much.

Only one thing, I had the structure wrong in the first place, so I deleted all the categories and tables in comppermine and started again. However, timer.php re-creates the cats and albums, - but then skips all the file uploads because it says they already exist!

I've even tried uploading a filesystem with different folder names, - and then overwriting it with my original one, but it still thinks the files exist.

Any ideas?
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on February 01, 2006, 03:08:09 am
If you only deleted the categories & tables, you didn't delete the pictures, did you?  You need to delete the categories & albums & pictures to start from scratch, then use timer.php to do its thing.  Then in the future, you add files & folders and run timer.php to add only the new ones.  Make sure you don't rename any categories or albums or timer.php will think the folder-named categories & albums don't exist and it will recreate them.  That's my main plan for a plugin; to take care of renames in Coppermine.

Also make sure you don't add folders in such a way that a previous album becomes a sub-category, if that makes sense.  You have to keep the levels straight since there are no sub-albums.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 01, 2006, 03:14:55 am
Ahh. Yes, it all makes sense, however, I now can't delete the pictures because I have a beautiful category structure - with no pictures in it. However, the main page footer tells me that there is 1600 pictures in 106 albums and 43 categories.

Somewhere, the application is holding info on the original pictures and thinks they are still there? (I suppose).

???
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on February 01, 2006, 03:31:18 am
Well I assume you emptied the tables directly in phpMyAdmin?  If so, you did something Coppermine cannot count on since it would never do that itself.  In effect, you hacked the database.  Empty the album table, the category table, and the pictures table, then try again.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 01, 2006, 03:44:23 am
You are correct. I did end up deleting categories using phpmyadmin. However, I did this because when I tried to do it in coppermine, they were locked and couldn't be deleted. Sorry, I can't remember the exact error message.

I have just deleted everything from the 3 databases as you suggest, and it's worked a charm. I appreciate your patience. You're a star!

This is an awesome mod. Thanks.

H.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on February 01, 2006, 03:47:06 am
Nibbler's the star for the mod; you're welcome for the support. 

I find this mod extremely useful as well and hope to make it more robust some time.  Others are welcome to do it instead or in addition - my toes are tough.

Oh, since some words were incorrectly used in the past few posts, here's a quick overview:
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 01, 2006, 02:59:49 pm
That's really useful info. And thanks again to Nibbler!   ;D

I am only having one slight problem now; timer stops after a few minutes (appears to crash). Not a hang, but the browser just goes to the 'no webpage found, - check dns setting etc), error.

I don't suppose there's a method of logging the script so that I can see which file/s causes this is there?

Logging would also be good to give a list of files it fails to load, so that I can manually convert formats etc. (just a suggestion).

Thanks again.

H.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on February 01, 2006, 05:04:34 pm
Depending upon your webserver's PHP settings, this is probably a PHP timeout issue. There are other threads that discuss this.  If you cannot adjust your PHP settings, then you have to work with what you have.

This script has a delay between requests ("sleep betwen additions") and a number of images to run on each page load ("limit additions per refresh").  Adjust these until things work and don't timeout.  Safe values I would guess would be a not-so-long delay (500-1000 ms) and a small number for refresh (5 or 10), depending upon how long it takes to create each thumbnail & intermediate image.

Yes, it would be good to log the operation of this script, ideally to the regular Coppermine log.  There are probably a few lines you could add to do that now, but I don't have time right now to check that.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 01, 2006, 07:05:12 pm
Thanks again for the reply.

Well, I've adjusted my timeouts in php.ini to 600 and also the filesize to 8m.
I've also changed my apache conf file timeout from 120 to 600. - All to no avail.

I wonder if (as a temp workaround), I can log into coppermine and upload the files in the empty directories using the standard coppermine interface. Or will this confuse things the next time I run timer.php?
 ???
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on February 01, 2006, 07:11:17 pm
Weird.  I don't know why you are having trouble.

Yes, you can use any of Coppermine's upload tools.  Timer.php uses Coppermine's table to figure out if the picture already exists.  As long as you use the same structure in both category & albums versus the folder structure, you can use either timer.php or Coppermine to add files and both will be happy.  In fact, timer.php uses Coppermine's functions to add everything; it just gathers things up at the beginning to automate the process you would do manually.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 01, 2006, 07:39:11 pm
Tis wierd isn't it. Even more so because I run timer.php with a completely different directory structure and it works fine!

Good that I can  do a few manual uploads though. I'll pick out the largest files in each directory first.

Couple of Q's, - when you have time !! :-

1/ Is it possible to 'break' the timer script if I set an album to be read only by a specific group?

2/ Can it crash if a file exceeds the max width/heigh in cpg config?

3/ Are there any known filetypes which cause conflicts? The directory structure I am testing might have some unsupported filetypes, whether by GD2 or coppermine.

4/ Is there also a timeout setting in MySQL5 which might need tweaking?

TIA, - as always     ;)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on February 01, 2006, 08:27:04 pm
This is one reason we should use a separate board for this one add-on.  I'll put together a pre-1.0 release plugin for this script and post it so we can start a sub-board on http://cpg-contrib.org/board/ (http://cpg-contrib.org/board/).  (Donnoman mentioned this earlier in this thread.)

One thing to note is that timer.php does not check for problematic characters in your directories.  Check for apostrophes or other "forbidden" characters (as noted in the coppermine config setting).  If you can figure out why one directory structure works but the other doesn't, please post because that will probably be useful for developing this script further.

I don't have any answers to your questions off the top of my head.  I'll check into it later when I have time, if no one else answers.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 01, 2006, 10:27:06 pm
Thanks. I promise to post back here too.

 8)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 02, 2006, 07:28:28 pm
And here I am   ;)

There is a problem i've discovered. Although the script creates albums under categories and sub categories in the correct way, the albums in coppermine are not classified under any category when you look at their properties.

As a result, normal http uploads are very difficult to assign to the correct album and if there are different albums with the same name in different directories, it is impossible to tell (from the album list in uploads, anyway), which are which.

H.
 
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 06, 2006, 09:17:39 am
Hi.
I have discovered another problem:

After I have run timer.php, I cannot remove any of the albums it created. I have tried to remove an album in coppermine, - so that I don't confuse the databases), before I delete the matching folder in my directory tree.
However, when I try to remove the album in cpg, I get an error message telling me that I don't have write access to it. Although, If I delete any other albums which I made in the standard way, (in coppermine), I have no problems.

I am running Apache 2 with mysql5 on a win2k installation. My root folder (under 'albums'), is set as write permissions for everyone, in 'sharing'.

I would certainly appreciate any pointers   ???
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on February 07, 2006, 04:14:29 am
timer doesn't modify the file permissions. It's probably a result of how the images were originally placed in the albums directory.

make sure all folders have been chmodded to 777
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: donnoman on February 07, 2006, 07:41:51 am
I've created a plugin conversion of timer.php called CPGMassImport.

you can find more information about it on cpg-contrib.org

http://cpg-contrib.org/board/index.php?topic=123.0

The new version is downloadable from sourceforge, check the link above for more information.

It will also be available via CVS.

Please keep the discusions of CPGMassImport on the cpg-contrib forums so as not to make this thread any more confusing than it already is.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: howie10 on February 14, 2006, 05:56:58 pm
timer doesn't modify the file permissions. It's probably a result of how the images were originally placed in the albums directory.

make sure all folders have been chmodded to 777

This is a windows 2k machine (Apache2). I can't chmod anything. I've set all folders and subdirectories to be writeable. Weird.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: redlabour on March 06, 2006, 06:48:36 pm
I dont understand this Thread - and i read it completely.

All i want to do is :

Move all Pictures and Categories from MKPortal/modules/coppermine/albums to /gallerie/albums with the same Filenames etc.

I uploaded the timer.php to my /gallerie/ and started it and nothing happens.

Only
Quote
Access denied
- no GUI - nothing.

Where is the Instruction for this Importer ?


Update :

Will use this one : http://cpg-contrib.org/board/index.php?topic=123.0
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: ktwenrick on April 30, 2006, 10:04:48 am
when i try and use it in a fresh install of 1.4.2 every photo fails..
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Joachim Müller on April 30, 2006, 10:55:42 am
don't use cpg1.4.2, it's outdated. Post a meaningful error message, link and more details. Just saying "doesn't work" is not helpful for supporters at all.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: ktwenrick on May 06, 2006, 02:36:25 am
ok thanks! the only reason i was using it was because my fantastico had it as the basic install.. i've now upgraded..thank ya
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: ktwenrick on May 06, 2006, 02:58:16 am
so since this mod is for 1.4.3 i'm guessing it wont work with 1.4.5??  or is there an upgrade anywhere? i have over 2500 pictures to upload and i'm not looking forward to it.. this batch upload would be great!

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Paver on May 06, 2006, 03:22:16 am
This thread is marked as 1.4.x so it should be fine.  I'm using timer.php fine with 1.4.5 (although I have modified it for my needs although not since 1.4.2).

However, you should consider using the new MassImport plugin which is based on timer.php.  See donnoman's post above and the link: http://cpg-contrib.org/board/index.php?topic=123.0 (http://cpg-contrib.org/board/index.php?topic=123.0).  (I am considering using it as well but I need to incorporate my mods to timer.php into the plugin so I'm waiting until I have time to do so.)
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: ktwenrick on May 07, 2006, 06:22:43 am
i was having a problem where out of no where my photos would fail to add.. but i just uninstalled the mod and reinstalled it and it stopped failing.. incase someone else has this problem.. hope that helps
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: mylogon on March 12, 2007, 11:01:01 am
Quote:

[Note: Tagged as 1.4, see later posts in thread for updated versions of this mod]

Why not put in in the first entry? Makes life simpler...

Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Joachim Müller on March 12, 2007, 12:47:10 pm
This mod has been made for cpg1.4.x, so it should work for any version (cpg1.4.0, cpg1.4.1, cpg1.4.2, cpg1.4.3, cpg1.4.4, cpg1.4., cpg1.4.5, cpg1.4.6, cpg1.4.7, cpg1.4.8, cpg1.4.9, cpg1.4.10), as suggested above. Since the cpg1.4.x series is the most recent stable one, there's no need to change anything imo. However, it's mandatory that you don't run outdated version, so your version should be cpg1.4.10. If you have an older version: upgrade. If this mod doesn't work with cpg1.4.10, please post details.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: UncleRemus on September 11, 2007, 11:28:25 pm
I just wanted to add that I got the script in the first post to work by adding "cpg_" before each call to db_query.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: Joachim Müller on October 02, 2007, 09:09:42 am
[Note: Tagged as 1.4, see later posts in thread for updated versions of this mod]
I took Flux's timer.php and merged a couple of my changes to his code.  This is tested with cpg 1.3, for 1.4 you should just need to change the db_query's with cpg_db_query's.
Title: Re: Batch add with automatic album/cat structure from directory tree.
Post by: ajbarickman on August 12, 2009, 07:53:21 pm
Does anyone know how to modify this script so that it will create albums NOT in categories.  I want to create a directory under albums and then have an album in no category created matching the folder name?

Any help would be greatly appreciated.