Broken file when uploading it in chunks via the API

Hi,
I use the Attachment API to upload files to a LiquidFiles server. This so far works fine. For big files, however, I run into memory issues with my current implementation, which is why I wanted to change the upload process, to upload the file in chunks. Doing so, however, resulted in broken files being uploaded, where the problem lies, I believe, in the wrong charset being used.

Correct:

Code:

%PDF-1.5
%ÐÔÅØ
90 0 obj
[...]

Broken:

Code:

%PDF-1.5
%Ã<90>Ã<94>Ã<85>Ã<98>
90 0 obj
[...]

My question now is, whether this could be an error on the LiquidFiles side. I use a perl script, which uses HTTP::Request::Common to do all the file processing, I don't do much at all. If I rebuild the content of the request and just write it to a file, everything is intact. I will provide a minimal example to upload files. Changing the variable $DYNAMIC_FILE_UPLOAD toggles the chunking. All suggestions are appreciated.

Code:

#!/usr/bin/perl -w

use strict;

use LWP::UserAgent qw( );
use HTTP::Request::Common qw($DYNAMIC_FILE_UPLOAD);
use MIME::Base64;

# Here the chunking can be enabled, which breaks certain files
$DYNAMIC_FILE_UPLOAD = 1;

# Provide Data
my $APIKey = 'API Key';
my $LiquidFileServer = 'https://my.lfserver.com';
my $File = '/path/to/file.ext';

my $KeyEncoded = encode_base64( "$APIKey:x\n" );
my $LFAuth = "Basic $KeyEncoded";

my $ua = LWP::UserAgent->new();
$ua->ssl_opts(verify_hostname => 0);
$ua->default_header( Authorization => $LFAuth );

# Upload File
my $request = HTTP::Request::Common::POST(
   "$LiquidFileServer/attachments",
   Content_Type => 'form-data',
   Content => [
      Filedata => [ $File ],
   ],
);

my $response = $ua->request($request);

my $AttachmentID = $response->content;

# Get URL
$response = $ua->post(
    "$LiquidFileServer/message",
    Content_Type => 'application/json',
    Content =>
'{"message":
  {
    "recipients":["foo@bar.com"],
    "subject":"test subject",
    "message":"Please let me know what you think!",
    "send_email":false,
    "authorization":0,
    "attachments":["'.$AttachmentID.'"]
  }
}'
);

$response->content =~ /url":"(.+)"/;
print "DownloadURL: $1\n";

 

Thanks Johan,
I might switch to using curl, I would rather give it one or two trys with HTTP::Request::Common before, however. The actual upload is part of a bigger program, where the latter is already included.

For the memory issue, you are right when it comes to normal usage. The LWP::UserAgent loads the whole file, and to me it seems not efficient in doing so, even a minimal upload script, which doesn't even catch the response, uses almost four times of the file size as memory, when run via valgrind --tool=massif. The chunked upload however works just fine in keeping the memory usage low (only a fraction of the file itself). The file is uploaded to LiquidFiles, too, it just breaks in the process...

The two upload versions work as follows:
In normal mode ($DYNAMIC_FILE_UPLOAD = 0), the file is loaded into a filehandle (see https://metacpan.org/source/OALDERS/HTTP-Message-6.18/lib/HTTP/Request/Common.pm#L156 for reference), in the next line binmode is set for the filehandle, and later (line 166) the whole content is slurped into a variable and somewhen sent as part of the content of the request. For small files this works fine, nothing breaks.

In the chunk mode ($DYNAMIC_FILE_UPLOAD = 0), instead of a content-string, a subroutine is given as content, which, as far as I understand it, is called again and again by the server, until nothing is returned anymore. The subroutine reads only a part of the file every time it is called, and returns it. (For reference, see https://metacpan.org/source/OALDERS/HTTP-Message-6.18/lib/HTTP/Request/Common.pm#L233 - this is initiated by $DYNAMIC_FILE_UPLOAD = 1.) In the subroutine the filehandle is defined in line 250, and here also, binmode is set in the next line.

As it is hard to exclude other errors on the perl side by just this, I wrote a small script (I will again include it), which just produces the request for the chunked upload method, and then calls the content-subroutine itself, to rebuild the file (with the boundary beginning and filename stuff in front; pipe the output into some outputfile, if you try it). This gives back the right content. So the error must be introduced somewhere in the transfer process, or maybe on the LiquidFiles side, in my opinion. But then again, I might be overlooking something, in any case, I don't really know where to look further.

Code:

#!/usr/bin/perl -w

use strict;

use LWP::UserAgent qw( );
use HTTP::Request::Common qw($DYNAMIC_FILE_UPLOAD);

# provide
my $File = "/path/to/file.ext";

$DYNAMIC_FILE_UPLOAD = 1;
my $ua = LWP::UserAgent->new();
my $request = HTTP::Request::Common::POST(
  'https://example.server/attachments',
   Content_Type => 'form-data',
   Content => [
      Filedata => [ $File ],
   ],
);

#optional
#use Data::Dumper;
#print Dumper($request);

my $contentsub = $request->content();

my $part = $contentsub->();
while ($part){
 print $part;
 $part = $contentsub->();
}