Advanced search  

News:

cpg1.5.48 Security release - upgrade mandatory!
The Coppermine development team is releasing a security update for Coppermine in order to counter a recently discovered vulnerability. It is important that all users who run version cpg1.5.46 or older update to this latest version as soon as possible.
[more]

Pages: [1]   Go Down

Author Topic: Large File support?  (Read 6004 times)

0 Members and 1 Guest are viewing this topic.

dcbrit2003

  • Coppermine newbie
  • Offline Offline
  • Posts: 11
Large File support?
« on: May 08, 2006, 10:14:35 pm »

Hi,

I'm currently trying to get CPG to add some video files to the database ( i.ve already uploaded them to my web server ).  The files range between 1.2 - 2.5 GB in size. I keep getting the message "no new file was found". I have checked and re-checked that I have used the correct syntax for the thumbnail image, i.e.:

file1.avi corresponds to thumb_file1.jpg

Initially I thought that it was Apache that was the problem as I was running ver 1.3.34 and so I installed ver. 2.2 which has Large File support. When I tried to access the files just via Apache ( i.e. from a basic directory and not using PHP, etc ) it now worked, whereas before I just got an error. I went back to trying to add the AVI files to the CPG database via the "batch add files" and still it was the same old message "no new file was found" yet the syntax for the files and thumds was fine. I then looked around to see if PHP was the problem and compiled in the following CFLAGS into PHP :

-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

I thought again that this would do the trick yet i'm no nearer solving this, it would seem. Can CPG handle files which have already been uploaded which are greater than 1GB in size? what else can I try?

p.s. I tried establishing beyond doubt that it is a file size issue buy copying over a 500mb file from another directory and renaming it it to the exact same filename in the exact same directory of the 1GB+ file that I was trying to add. This worked fine and CPG had no problem adding the smaller file so it is a file size issue

Regards, Mark
« Last Edit: May 11, 2006, 10:14:14 pm by GauGau »
Logged

dcbrit2003

  • Coppermine newbie
  • Offline Offline
  • Posts: 11
Re: Large File support?
« Reply #1 on: May 09, 2006, 11:35:18 am »

i'm using CPG 1.4.4, PHP 5.1.4 Apache 2.2.2 and MySQL 5.0.20a

Any help will be greatly appreciated
Logged

Nibbler

  • Guest
Re: Large File support?
« Reply #2 on: May 09, 2006, 03:07:29 pm »

I don't have any files of that size to test with. It's more likely to be an underlying issue with server setup than anything directly related to Coppermine since the script only use the functions built-in to php to read and manipulate files
Logged

dcbrit2003

  • Coppermine newbie
  • Offline Offline
  • Posts: 11
Re: Large File support?
« Reply #3 on: May 09, 2006, 05:20:00 pm »

Hi,

I am using Apache 2.2.2 which has Large File support and I have compiled in Large File support to PHP 5.1.4. I did a test by replacing an existing movie file ( size of about 350MB ) which was succesfully added to the CPG database, with one of the 1GB+ movie files by using the exact same filename in the exact same directory. Unfortunetly, the encoding used in the two files is different ( WMV vs AVI ) so all I could expect was my browser to load the Windows Media Player Plugin and then just sit there donig nothing. This is exactly what happened so it is obviously not a filesystem, Apache or PHP issue. It seems to be a CPG issue ( From Apache log )

:xx.xxx.xx.xx - - [09/May/2006:10:04:00 -0500] "GET /members/albums/movies/512k/gdfgdf/dfgdfgdf-512k.wmv HTTP/1.1" 200 2256795648

The above request was done from within the PHP interface so large files are not a problem with my PHP + Apache installation yet when I try adding 1GB+ files to my CPG installation's database I cannot.

Regards, Mark

Logged

Joachim Müller

  • Dev Team member
  • Coppermine addict
  • ****
  • Offline Offline
  • Gender: Male
  • Posts: 47843
  • aka "GauGau"
    • gaugau.de
Re: Large File support?
« Reply #4 on: May 09, 2006, 11:19:31 pm »

what happens if you try to load the file using your media player directly (using the player's "open" dialog), i.e. without embedding the player into your browser?
Logged

dcbrit2003

  • Coppermine newbie
  • Offline Offline
  • Posts: 11
Re: Large File support?
« Reply #5 on: May 10, 2006, 07:11:36 pm »

I just tried this and it downloads the file :

xx.xxx.xx.xx - - [10/May/2006:12:04:25 -0500] "GET /members/albums/movies/3000k/fghgfhfghfg/fghfghfg.avi HTTP/1.1" 200 3063927980

so i'm just wondering why CPG has a problem with large ( 1GB+ ) files.

Regards, Mark
Logged

Joachim Müller

  • Dev Team member
  • Coppermine addict
  • ****
  • Offline Offline
  • Gender: Male
  • Posts: 47843
  • aka "GauGau"
    • gaugau.de
Re: Large File support?
« Reply #6 on: May 10, 2006, 07:26:59 pm »

Coppermine is a PHP script and therefor bound to any limitations set up on the server for PHP. As a guess, I'd look at the execution time / time-out settings in php.ini (if the server is yours to administer). There's no explicit code in Coppermine (at least none that I'm aware of) that artificially limits the size of a video file, that's why my guess is a PHP limitation.
Logged

dcbrit2003

  • Coppermine newbie
  • Offline Offline
  • Posts: 11
Re: Large File support?
« Reply #7 on: May 11, 2006, 03:08:57 am »

I think that the solution here is to just split my 1GB+ movies up. This will be a PITA because i'll have to re-encode them and upload them ( all 32 GB's worth ) but I don't see any other way.

Logged

Paver

  • Dev Team member
  • Coppermine addict
  • ****
  • Country: us
  • Offline Offline
  • Gender: Male
  • Posts: 1609
  • Paul V.
Re: Large File support?
« Reply #8 on: May 11, 2006, 03:31:11 am »

There might be a way to split the files byte-wise, upload, then put them back together again on the server.  I know you can do that with shell access, the question is can you do it in PHP and create a Coppermine add-on/mod/plugin to do this for you.

I know some people have considered this in threads on this forum before, but I don't remember what or if they decided anything.
Logged

Tranz

  • Dev Team member
  • Coppermine addict
  • ****
  • Country: 00
  • Offline Offline
  • Gender: Female
  • Posts: 6149
Re: Large File support?
« Reply #9 on: May 11, 2006, 05:06:56 am »

If it's a pain to upload huge files, it'll be a pain to download them, too...
Logged

Paver

  • Dev Team member
  • Coppermine addict
  • ****
  • Country: us
  • Offline Offline
  • Gender: Male
  • Posts: 1609
  • Paul V.
Re: Large File support?
« Reply #10 on: May 11, 2006, 05:13:42 am »

Heh.  That's a really good point. 
Logged

Joachim Müller

  • Dev Team member
  • Coppermine addict
  • ****
  • Offline Offline
  • Gender: Male
  • Posts: 47843
  • aka "GauGau"
    • gaugau.de
Re: Large File support?
« Reply #11 on: May 11, 2006, 09:43:59 am »

I fully and whole-heartedly agree. I can't see any good reason why people should download videos that "weigh" Gibabytes or more using their browser (i.e. Coppermine) ::). If you must distribute such large files, use FTP or file sharing like BitTorrent - anything that allows downloads to be resumed. Even users on broadband will have issues downloading such huge files, no matter what the actual content may be.
You better think about codecs that have better compression (like DivX or similar). 32 GB for one video file just is a bit too much.
Logged

Paver

  • Dev Team member
  • Coppermine addict
  • ****
  • Country: us
  • Offline Offline
  • Gender: Male
  • Posts: 1609
  • Paul V.
Re: Large File support?
« Reply #12 on: May 11, 2006, 03:58:14 pm »

You could use Coppermine to organize the .torrent files if you have a need for 32 GB video files.
Logged

dcbrit2003

  • Coppermine newbie
  • Offline Offline
  • Posts: 11
Re: Large File support?
« Reply #13 on: May 11, 2006, 09:35:18 pm »

To answer several points above:

I don't have any files which are 32GB in size, someone seems to be guilty of not reading my posts. As i've said, all my video files are 1.2 - 2.5 GB in size. it's a PITA to split them up using VirtualDub and then re-upload them because my upload speed is restricted to about 300Kb/s, whereas my download speed is 10 Mb/s so it would not be much of a pain for people to download them. My movies are already encoded using Divx.

Regards, Mark
Logged

dcbrit2003

  • Coppermine newbie
  • Offline Offline
  • Posts: 11
Re: Large File support?
« Reply #14 on: May 11, 2006, 09:46:10 pm »

There might be a way to split the files byte-wise, upload, then put them back together again on the server.  I know you can do that with shell access, the question is can you do it in PHP and create a Coppermine add-on/mod/plugin to do this for you.

I know some people have considered this in threads on this forum before, but I don't remember what or if they decided anything.

I used an MSDOS version of the unix 'split' command to split the 1GB+ movie files up before uploading and re-combine them using 'cat' to reconsitute the files after uploading before I realised I had this Large File problem. I used two computers on the same network to upload each piece to minimise any data corruption problem stoping the whole upload process, which sometimes happens on my SFTP client. Now i'll have to split each file into seperate video files ( Part 1, 2,3, etc ) so as not to be more then 750MB just to be on the safe side.

Regards, Mark
Logged

Paver

  • Dev Team member
  • Coppermine addict
  • ****
  • Country: us
  • Offline Offline
  • Gender: Male
  • Posts: 1609
  • Paul V.
Re: Large File support?
« Reply #15 on: May 11, 2006, 10:21:52 pm »

I assume you are using something like MD5 checksums to verify each part and the final file, but in case not, I thought I'd mention it.  RFSync is a good tool to fix errors when found.
Logged

Joachim Müller

  • Dev Team member
  • Coppermine addict
  • ****
  • Offline Offline
  • Gender: Male
  • Posts: 47843
  • aka "GauGau"
    • gaugau.de
Re: Large File support?
« Reply #16 on: May 11, 2006, 10:37:28 pm »

I don't have any files which are 32GB in size, someone seems to be guilty of not reading my posts.
I confess that I'm going to be guilty to ignore your postings from now on... Sorry for misreading your important posting, this can happen if you read thousands of postings ::) In the end, 1 GB or 32 GB doesn't make a difference to me: such files are too large for http
Logged
Pages: [1]   Go Up
 

Page created in 0.024 seconds with 20 queries.