x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

[HTTP] Why is HTTP GET so slow, UE4 is performing 15x slower than Python at downloading a file

Dear Epic and Community,

Does anyone know a way to speed up the use of HTTP GET requests on windows machine?

I can download 1.67 GB in about 1.5 min using chrome or my ftp server, but using HTTP GET request from within UE4 takes at least 30 minutes.

Why is it literally about 20x slower?

For comparison I downloaded the 1.67 gb file in 2 min and 20 seconds using python, which is about the same as Chrome and my FTP server

I really wish UE4 could download things faster for me than python can :)

As more and more UE4 projects are maturing and needing fast download speeds for DLC and paks and all sorts of things, I really think fast download speeds should be a high priority :)

My research so far:

I feel like libcurl is capped somehow in a way it shouldnt be.

I tried turning lib curl off but then the download does not occur at all. (I tried both config and commandline methods)

I tried accessing the public static FCurlHttpManager ptr and including the private .h file, but the compiler was not finding it even with HTTP module included

 static CURLSH* GShareHandle;

Even if I did succeed in that I wouldnt really know what to do to make UE4 download things faster.

Is there anyway to make HTTP GETs go faster on a windows machine?

Relevant code can be found in AsyncTaskDownloadImage.cpp, this is what I am doing, but for a 1.67 gb file.

 TSharedRef<IHttpRequest> HttpRequest = FHttpModule::Get().CreateRequest();
 
 HttpRequest->OnProcessRequestComplete().BindUObject(this, &UAsyncTaskDownloadImage::HandleImageRequest);
 HttpRequest->SetURL(URL);
 HttpRequest->SetVerb(TEXT("GET"));
 HttpRequest->ProcessRequest();

I wrote my own download progress bar using

 //IHttp
 /**
  * Delegate called to update the request/response progress. See FHttpRequestProgressDelegate
  */
 virtual FHttpRequestProgressDelegate& OnRequestProgress() = 0;

So I know the download is working, it just takes a really really long time compared to Chrome or my FTP server.

Thank you for any insights in this matter!

A Neutral Test ~ 1GB Speed Test

If you want to test you can test with a 1 GB download here:

ftp://speedtest:speedtest@ftp.otenet.gr/test1Gb.db

I endured it, in UE4 it took:

17 minutes and 43 seconds

Exact same url, using python:

1 minute and 19 seconds

My Conclusion

Clearly libcurl is throttled somewhere, and cannot use the full power of my internet connection, in a way that Chrome, my ftp server, and python are not limited.

Can you please address this so I can use UE4 instead of python to download DLC?

Thank youuuu!

Rama

PS: Here is the python script I used:

 import urllib2
 
 downloadedfile = urllib2.urlopen("http://YourVeryLargeFile.exe")
 with open('pO.exe', 'wb') as output:
     while True:
         data = downloadedfile.read(100000000)
         if data:
             output.write(data)
             print "read 100 mb"
         else:
             print "download complete and file written!"
             break





Product Version: UE 4.12
Tags:
more ▼

asked Jul 22 '16 at 05:51 AM in C++ Programming

avatar image

Rama
10.5k 443 340 1078

avatar image NoobsDeSroobs Jul 22 '16 at 07:39 AM

Sorry for being pedantic, but is it 15 or 20 times slower? The title and the body are conflicting.

avatar image Rama Jul 22 '16 at 07:45 AM

UE4 http get is 15 x slower than python, and 20x slower than Chrome/my ftp server.

Also please see my neutral 1gb speed test which I just posted:

UE4 17 minutes 43 seconds

python 1 minute 19 seconds

I performed these test 1 right after the other, and not concurrently, same url :)

Chrome, my ftp server, and python are all very similar, and UE4 http get using libcurl is at least 15x slower at a minimum.

avatar image Moss Jul 23 '16 at 05:08 PM

Might be the buffering, in your python script you just read 100000000 bytes while Epics code is buffering to throttle the download.

avatar image Rama Jul 25 '16 at 10:01 PM

Hi Moss!

Very nice to hear from you!

Rama

avatar image cageman Aug 30 '16 at 09:01 PM

An important note for this issue. On windows we can switch to WinInet using the following code in defaultengine.ini:

 [Networking]
  UseLibCurl=False

The problem is that Linux only has LibCurl and its way too slow. WinInet did work for my purposes though and is very fast. Another note: i tested LibCurl using the command line of LibCurl and all my calls were very fast as well, so i suspect it has something to do on how UE4 is using libcurl.

I figured out i actually had one special call in which i had a get request with an empty json as input (using varest plugin), apparently that caused the long http calls. It is somehow related to libcurl..

(comments are locked)
10|2000 characters needed characters left

3 answers: sort voted first

Hey Rama,

Thanks for the feedback. I was able to reproduce the issue and it is submitted as, UE-33732.

more ▼

answered Jul 25 '16 at 06:47 PM

avatar image

ImVawx ♦♦ STAFF
6.7k 114 16 126

avatar image Rama Jul 25 '16 at 10:00 PM

Thank you for the update Kyle!

Rama

avatar image burak.io Sep 14 '16 at 11:11 AM

Is there any temporary solution for Android and IOS to get normal speed before this gets fixed?

avatar image ImVawx ♦♦ STAFF Sep 14 '16 at 01:24 PM

There hasn't been any update on a current work around or fix for this issue but another user has suggested something that may or may not work. You can see that here:

https://answers.unrealengine.com/questions/320627/varest-http-get-slow.html

avatar image cageman Sep 14 '16 at 05:21 PM

For Android and IOS there is currently no workaround. The suggested workaround only works on windows, since there is support for WinInet there, which actually works fast. I am still suprised this is currently backed logged, how can you possibly be using HTTP with this speed? I consider it completely broken at the moment.

avatar image ImVawx ♦♦ STAFF Sep 14 '16 at 05:27 PM

The only note on it is that it isn't affecting anything bad enough to be critical. There is a plan to look at it whenever they can get to it, though.

(comments are locked)
10|2000 characters needed characters left

This is already marked as resolved question, but I want to share I discovered. I found that HTTP request's fetch rate is controlled by FHttpModule, especially HttpThreadActiveFrameTimeInSeconds variable is the key. Default value of it is around 0.005(200Hz). That means, tiny fetch operations were called just 200 times per second. By decreasing the value to the proper value I could get reasonable download speed.

You can control this value from the Engine configuration file like this:

 [HTTP]
 HttpThreadActiveFrameTimeInSeconds=0.00001
more ▼

answered Nov 20 '17 at 09:33 AM

avatar image

nullbus
6 1 4 8

(comments are locked)
10|2000 characters needed characters left

Along the lines of nullbus's answer, you can increase the buffer size to the libcurl maximum allowed value. For a large download over a very strong internet connection, this dropped my response times from ~8.3 seconds to ~1.3 seconds.

 [HTTP.Curl]
 BufferSize=524288

more ▼

answered Feb 11 '19 at 02:55 PM

avatar image

Lucas09090909091
1

(comments are locked)
10|2000 characters needed characters left
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question