Sending a file under /usr with RequestHttpx

Switch to desktop version

Sending a file under /usr with RequestHttpx - Printable Version

+- Ewon Technical Forum (https://techforum.ewon.biz)
+-- Forum: Development (https://techforum.ewon.biz/forum-50.html)
+--- Forum: Java (https://techforum.ewon.biz/forum-53.html)
+--- Thread: Sending a file under /usr with RequestHttpx (/thread-749.html)

Sending a file under /usr with RequestHttpx - Patrick.F - 19-10-2018

Hello,

My firmware : 13.0s0
My eWON : flexy 205

I try to send a file to a Google Cloud bucket in Java with RequestHttpX() but unfortunately the file is always empty. Sometimes the returen status is 0 (success) and sometimes it is 32603.

So I decided to try to send the file to the eWON test server 'ewonsupport.biz/post' and I have the same behavoir.

Here is my code with the eWon server :

Code:
status = ScheduledActionManager.RequestHttpX("https://ewonsupport.biz/post/post_example.php", "POST", "", "send=first", "file=[$dtUF $uf/usr/photo010.jpg $fnp1.jpg]", fileAnswer);

System.out.println("RequestHttpX: status="+status);

-> RequestHttpX: status=32603

I even try with PutHttp() but I have the same symptom:

Code:
int status2 = ScheduledActionManager.PutHttp("https://ewonsupport.biz", "/post/post_example.php", "send=second", "file=[$dtUF $uf/usr/photo010.jpg $fnp2.jpg]", "POSTERROR");

System.out.println("PutHttp: status="+status2);

-> PutHttp: status=32603

I cannot figurate where I am wrong.

Need help.
Thanks,

Patrick

Other question :
In the file field, I use "file=...". Can I put any name ? What it is for ?

Thanks

RE: Sending a file under /usr with RequestHttpx - Patrick.F - 19-10-2018

After multiple tries, I can say I have no coding error but file size limitation !
It seems that I cannot send a file greater than 10 KB: I got the status 32603 and not file is received from google cloud side.
For file greater than 8 KB, the file is received from google cloud side but I still have the status 32603 !

Anyway 10 KB limitation is a big issue but not the last.

In fact RequestHttpX builds a multipart message and the file content format is 'Content-Type: application/octet-stream' and unfortunalely this content is not supported by google cloud.

Is there any way to change the content format ? Or to not use multipart format.

Thanks.

PS: Here is the beginning what it is send by RequestHttpX:

-----------------------------7d624e177018a
Content-Disposition: form-data; name="file"; filename="photo4.jpg"
Content-Type: application/octet-stream

ÿØÿà JFIF ` ` ÿÛ C

RE: Sending a file under /usr with RequestHttpx - simon - 23-10-2018

Patrick,

Regarding the file size issue, can you try with the latest firmware 13.1PR available on https://websupport.ewon.biz/support/product/manual-firmware-update/manual-firmware-download ?
In this firmware, a similar issue has been fixed. I hope your case will be fixed too.

Regarding the header used, yes, you can customize the HTTP header in the 3rd parameter of the RequestHTTPX function :
ScheduledActionManager.RequestHttpX("https://ewonsupport.biz/post/post_example.php", "POST", "Content-type=application/json", ...

Simon

RE: Sending a file under /usr with RequestHttpx - lgn@ltr@n - 23-10-2018

Hello Simon,

My colleague Patrick is in vacation ... so I give some little more info.
I have not (yet) upgraded to new Firmware 13.1PR, and I used a 6 Kbyte image for this example to avoid known filesize issue.

We need to upload images to google bucket/cloud storage from an Ewon.

Choice 1: https://cloud.google.com/storage/docs/json_api/v1/how-tos/multipart-upload
Choice 2 : https://cloud.google.com/storage/docs/json_api/v1/how-tos/simple-upload

So far we didn't succeed in any of these choices.

for Choice 1, here I stand today:

       StringBuffer headersBuf = new StringBuffer();
       headersBuf.append("Authorization=Bearer ").append(accessToken);

       StringBuffer urlBuf = new StringBuffer();
       urlBuf.append("https://www.googleapis.com/upload/storage/v1/b/");
       urlBuf.append(bucketName);
       urlBuf.append("/o?uploadType=multipart/related");
       urlBuf.append("&name=").append(pictureName);


       StringBuffer fileFieldsBuf = new StringBuffer();
       fileFieldsBuf.append("file=[$dtUF $uf");
       fileFieldsBuf.append("/usr/index2.json");
       fileFieldsBuf.append(" $fn").append(pictureName).append(".json]");

       fileFieldsBuf.append("&file=[$dtUF $uf");
       fileFieldsBuf.append(fileName);
       fileFieldsBuf.append(" $fn").append(pictureName).append(".jpg]");

       status = ScheduledActionManager.RequestHttpX(urlBuf.toString(), "POST", headersBuf.toString(), "", fileFieldsBuf.toString(), fileAnswer);

this more or less builds the content that almost matches google requirements, but not fully.
I capture the POST with tcpdump (used a local server instead of google as target) and we can see
a couple of issues :

1/ the top-level Content-Type is "multipart/form-data" while Google expects "multipart/related"
2/ the two parts have also innacurate content type :
Content-Type: application/octet-stream
for google, should be :
Content-Type: application/json; charset=UTF-8 (for the JSON file)
Content-Type: image/jpeg (for the image)
3/ The file contents in the POST are OK (good news, we get the data we expected from the filesystem). but for json part should also be in plain text, as per google requierements.

So the question is how can we set the parts content type to match google expectations and actual MIME types of files we selected ?

Frame 67: 195 bytes on wire (1560 bits), 195 bytes captured (1560 bits)
Ethernet II, Src: ActL_46:4a:2a (00:03:27:46:4a:2a), Dst: HewlettP_61:37:da (e4:11:5b:61:37:da)
Internet Protocol Version 4, Src: 10.35.108.66, Dst: 10.35.108.77
Transmission Control Protocol, Src Port: 46593, Dst Port: 80, Seq: 6809, Ack: 1, Len: 129
[7 Reassembled TCP Segments (6937 bytes): #55(315), #57(187), #59(1448), #61(1448), #63(1448), #65(1962), #67(129)]
    [Frame: 55, payload: 0-314 (315 bytes)]
    [Frame: 57, payload: 315-501 (187 bytes)]
    [Frame: 59, payload: 502-1949 (1448 bytes)]
    [Frame: 61, payload: 1950-3397 (1448 bytes)]
    [Frame: 63, payload: 3398-4845 (1448 bytes)]
    [Frame: 65, payload: 4846-6807 (1962 bytes)]
    [Frame: 67, payload: 6808-6936 (129 bytes)]
    [Segment count: 7]
    [Reassembled TCP length: 6937]
    [Reassembled TCP Data: 504f5354202f75706c6f61642f73746f726167652f76312f...]
Hypertext Transfer Protocol
    POST /upload/storage/v1/b/test-eric-altran/o?uploadType=multipart/related&name=index2 HTTP/1.1\r\n
    Host: 10.35.108.77\r\n
    Connection: TE, close\r\n
    TE: trailers\r\n
    Authorization: Bearer toto\r\n
    Pragma: no-cache\r\n
    Content-Type: multipart/form-data; boundary=---------------------------7d624e177018a\r\n
    Transfer-Encoding: chunked\r\n
    \r\n
    [Full request URI: http://10.35.108.77/upload/storage/v1/b/test-eric-altran/o?uploadType=multipart/related&name=index2]
    [HTTP request 1/1]
    [Response in frame: 69]
    HTTP chunked response
        Data chunk (179 octets)
            Chunk size: 179 octets
            Data (179 bytes)
                Data: 2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d...
                [Length: 179]
            Chunk boundary: 0d0a
        Data chunk (6298 octets)
            Chunk size: 6298 octets
            Data (6298 bytes)
                Data: 2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d...
                [Length: 6298]
            Chunk boundary: 0d0a
        Data chunk (116 octets)
            Chunk size: 116 octets
            Data (116 bytes)
                Data: 6ba99e169b5b4828888f3bf6adbc388ad5ff000ae7ff004a...
                [Length: 116]
            Chunk boundary: 0d0a
        End of chunked encoding
        \r\n
    File Data: 6593 bytes
MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: "---------------------------7d624e177018a"
    [Type: multipart/form-data]
    First boundary: -----------------------------7d624e177018a\r\n
    Encapsulated multipart part: (application/octet-stream)
        Content-Disposition: form-data; name="file"; filename="index2.json"\r\n
        Content-Type: application/octet-stream\r\n\r\n
        Data (22 bytes)
            Data: 7b20226e616d65223a20226d794f626a65637422207d
            [Length: 22]
    Boundary: \r\n-----------------------------7d624e177018a\r\n
    Encapsulated multipart part: (application/octet-stream)
        Content-Disposition: form-data; name="file"; filename="index2.jpg"\r\n
        Content-Type: application/octet-stream\r\n\r\n
        Data (6212 bytes)
            Data: ffd8ffe000104a46494600010100000100010000ffdb0084...
            [Length: 6212]
    Last boundary: \r\n-----------------------------7d624e177018a--\r\n

The other question is why changing the top level content type breaks everything ???

I justd added 1 line of code to better match google's requirements on top headers :

        headersBuf.append("&Content-Type=multipart/related");

and the output is now :

Frame 31: 311 bytes on wire (2488 bits), 311 bytes captured (2488 bits)
Ethernet II, Src: ActL_46:4a:2a (00:03:27:46:4a:2a), Dst: HewlettP_61:37:da (e4:11:5b:61:37:da)
Internet Protocol Version 4, Src: 10.35.108.66, Dst: 10.35.108.77
Transmission Control Protocol, Src Port: 46597, Dst Port: 80, Seq: 1, Ack: 1, Len: 245
Hypertext Transfer Protocol
    POST /upload/storage/v1/b/test-eric-altran/o?uploadType=multipart&name=index2 HTTP/1.1\r\n
    Host: 10.35.108.77\r\n
    Connection: TE, close\r\n
    TE: trailers\r\n
    Authorization: Bearer toto\r\n
    Content-Type: multipart/related\r\n
    Pragma: no-cache\r\n
    Content-Length: 0\r\n
        [Content length: 0]
    \r\n
    [Full request URI: http://10.35.108.77/upload/storage/v1/b/test-eric-altran/o?uploadType=multipart&name=index2]
    [HTTP request 1/1]
    [Response in frame: 33]

there is no more files uploaded.

Please tell us what can be done better with choice 1. Looks to me there are several issues at least on the MIME types which should be better handled by Ewon.

Now about choice 2, what are our options ?

From my attempts, as soon as we set the top header content type to anything, the whole file information is skipped by requestHttpX().
So do we need to read the file by ourselves in JAVA and provided this as the body String in requestHttpX() ?

I had a quick look to https://techforum.ewon.biz/thread-217.htmlas an alternative, but it seems this will not work with HTTPS ?
AS you said :

Quote:If your Flexy/CD is connected to Talk2M, you can use the Custom HttpClient and the M2U service to convert HTTP to HTTPS (See https://developer.ewon.biz/content/https-m2u).
If not, you can use the functions com.ewon.ewonitf.ScheduledActionManager.GetHttp(...) or PutHttp(..) which support HTTPS but not the header customization.
If none of the solutions fit your application, let me know.

I guess we're exactly at this boundary .... none of the solution fits our application ... I'm not sure sending requests to google, with an Oauth bearer as identification through VPN/M2U is a good idea. this really adds complexity and one more service in the already complex stuff.

I'd really prefer that requestHttpX() helps us better formatting our requests so they can be accepted by google.

Thanks for your feedback.
Laurent

RE: Sending a file under /usr with RequestHttpx - lgn@ltr@n - 23-10-2018

Simon,

About choice 2 (https://cloud.google.com/storage/docs/json_api/v1/how-tos/simple-upload), I tried the following :

char[] cbuf = new char[8192];
try {
FileReader reader = new FileReader(fileName);
reader.read(cbuf);
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

String body = new String (cbuf);

System.out.println("cbuf size=" +body.length());
StringBuffer debug= new StringBuffer();
for (int idx=0; idx<30; idx++) {
debug.append(Integer.toHexString((int) body.charAt(idx)));
debug.append(" ");
}
System.out.println("cbuf=" +debug);
StringBuffer headersBuf = new StringBuffer();
headersBuf.append("Authorization=Bearer ").append(accessToken);
headersBuf.append("&Content-Type=image/jpeg");

StringBuffer urlBuf = new StringBuffer();
//urlBuf.append("https://www.googleapis.com/upload/storage/v1/b/");
urlBuf.append("http://10.35.108.77/upload/storage/v1/b/");
urlBuf.append(bucketName);
urlBuf.append("/o?uploadType=media");
urlBuf.append("&name=").append(pictureName);

StringBuffer fileFieldsBuf = new StringBuffer();

status = ScheduledActionManager.RequestHttpX(urlBuf.toString(), "POST", headersBuf.toString(), body, fileFieldsBuf.toString(), fileAnswer);

from RT logs, I can see I have a string containing the image payload

23/10/2018 14:51:05 JVM cbuf=ff d8 ff e0 0 10 4a 46 49 46 0 1 1 0 0 1 0 1 0 0 ff db 0 84 0 9 6 7 12 10 94406233 9321

but the RequestHttpX method will stop reading the buffer at first 0x0 in the string

resulting POST is thus incomplete ... I have only the first 4 bytes of the file

Frame 31: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ActL_46:4a:2a (00:03:27:46:4a:2a), Dst: HewlettP_61:37:da (e4:11:5b:61:37:da)
Internet Protocol Version 4, Src: 10.35.108.66, Dst: 10.35.108.77
Transmission Control Protocol, Src Port: 46622, Dst Port: 80, Seq: 235, Ack: 1, Len: 4
[2 Reassembled TCP Segments (238 bytes): #29(234), #31(4)]
Hypertext Transfer Protocol
POST /upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=index2 HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=index2 HTTP/1.1\r\n]
Request Method: POST
Request URI: /upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=index2
Request Version: HTTP/1.1
Host: 10.35.108.77\r\n
Connection: TE, close\r\n
TE: trailers\r\n
Authorization: Bearer toto\r\n
Content-Type: image/jpeg\r\n
Pragma: no-cache\r\n
Content-Length: 4\r\n
[Content length: 4]
\r\n
[Full request URI: http://10.35.108.77/upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=index2]
[HTTP request 1/1]
[Response in frame: 33]
File Data: 4 bytes
[Expert Info (Note/Malformed): HTTP body subdissector failed, trying heuristic subdissector]
Media Type
Media type: image/jpeg (4 bytes)
=> ff d8 ff e0

here is a totally valid POST done from postman, for comparison (using same image file as source)

Frame 4: 6765 bytes on wire (54120 bits), 6765 bytes captured (54120 bits)
Ethernet II, Src: Alcatel-_b4:43:c2 (00:e0:b1:b4:43:c2), Dst: HewlettP_61:37:da (e4:11:5b:61:37:da)
Internet Protocol Version 4, Src: 10.240.240.50, Dst: 10.35.108.77
Transmission Control Protocol, Src Port: 55823, Dst Port: 80, Seq: 1, Ack: 1, Len: 6711
Hypertext Transfer Protocol
POST /upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=ewonLolo-photo001 HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=ewonLolo-photo001 HTTP/1.1\r\n]
Request Method: POST
Request URI: /upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=ewonLolo-photo001
Request Version: HTTP/1.1
Content-Type: image/jpeg\r\n
cache-control: no-cache\r\n
Postman-Token: 95246ab7-c3ba-46a5-97fb-5fedd37f21c0\r\n
Authorization: Bearer toto\r\n
User-Agent: PostmanRuntime/7.3.0\r\n
Accept: */*\r\n
Host: 10.35.108.77\r\n
accept-encoding: gzip, deflate\r\n
content-length: 6212\r\n
[Content length: 6212]
Connection: keep-alive\r\n
\r\n
[Full request URI: http://10.35.108.77/upload/storage/v1/b/test-eric-altran/o?uploadType=media&name=ewonLolo-photo001]
[HTTP request 1/1]
[Response in frame: 6]
File Data: 6212 bytes
JPEG File Interchange Format
=> ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 84 00 09 06 07 12 10 12 15 12 12 12 15 15 15 17
....

So, what can I do now ?
we really need to upload images to google for our customer and it looks Ewon has major limitations on handling such uploads.

Rgds
Laurent

RE: Sending a file under /usr with RequestHttpx - lgn@ltr@n - 23-10-2018

Simon,

FYI, using my code toward real Google server with a valid bearer token, and a 33Kbytes Jpeg as source file, I get the following final status from google for choice 2 (simple upload)

status    "Ok"
additionalInfo
kind       "storage#object"
id            "test-eric-altran/jpeg2000/1540307555089576"
selfLink "https://www.googleapis.com/storage/v1/b/test-eric-altran/o/jpeg2000"
name    "jpeg2000"
bucket "test-eric-altran"
generation          "1540307555089576"
metageneration               "1"
contentType      "image/jpeg"
timeCreated      "2018-10-23T15:12:35.089Z"
updated               "2018-10-23T15:12:35.089Z"
storageClass       "REGIONAL"
timeStorageClassUpdated           "2018-10-23T15:12:35.089Z"
size        "4"
md5Hash             "0D2GS39D25zjTfX3IFCdDg=="
mediaLink           "https://www.googleapis.com/download/storage/v1/b/test-eric-altran/o/jpeg2000?generation=1540307555089576&alt=media"
crc32c    "JAabyQ=="
etag       "CKj5qeXsnN4CEAE="

the file is created and only has 4 first bytes, the rest of the file being lost by RequestHttpX()

for choice 1, as expected, google rejects the creation because of several issues in the Content-Type generated by RequestHttpX()

status   "Error"
errorInfo
error
errors
0
domain   "global"
reason   "badContent"
message   "Unsupported content with type: application/octet-stream"
code   400
message   "Unsupported content with type: application/octet-stream"

We're stuck tonight.

Rgds
Laurent

RE: Sending a file under /usr with RequestHttpx - lgn@ltr@n - 24-10-2018

Hello,

Just to make things clear,

issue is stille the same with firmware 13.1PR.

- We cannot send simple upload because RequestHttpX() ends the data at first null char in the 'String' that represents the JPEG
- We cannot use the multipart upload because it doesn't allow us to finely customize the Content Type (top level and file level) and json metadata is send as raw bytes while it should be text. (this is at least the things we identified as blocking).

Rgds
Laurent

RE: Sending a file under /usr with RequestHttpx - simon - 26-10-2018

Laurent,

I simulate the same and it seems to be a limitation of the RequestHTTPX function.
I will discuss internally if we can improve it to fit your needs.

Sorry for the inconvenience

Simon