[HTTP] Why is HTTP GET so slow, UE4 is performing 15x slower than Python at downloading a file

Dear Epic and Community,

Does anyone know a way to speed up the use of HTTP GET requests on windows machine?

I can download 1.67 GB in about 1.5 min using chrome or my ftp server, but using HTTP GET request from within UE4 takes at least 30 minutes.

Why is it literally about 20x slower?

For comparison I downloaded the 1.67 gb file in 2 min and 20 seconds using python, which is about the same as Chrome and my FTP server

I really wish UE4 could download things faster for me than python can :slight_smile:

As more and more UE4 projects are maturing and needing fast download speeds for DLC and paks and all sorts of things, I really think fast download speeds should be a high priority :slight_smile:

My research so far:

I feel like libcurl is capped somehow in a way it shouldnt be.

I tried turning lib curl off but then the download does not occur at all. (I tried both config and commandline methods)

I tried accessing the public static FCurlHttpManager ptr and including the private .h file, but the compiler was not finding it even with HTTP module included

static CURLSH* GShareHandle;

Even if I did succeed in that I wouldnt really know what to do to make UE4 download things faster.

Is there anyway to make HTTP GETs go faster on a windows machine?

Relevant code can be found in AsyncTaskDownloadImage.cpp, this is what I am doing, but for a 1.67 gb file.

TSharedRef<IHttpRequest> HttpRequest = FHttpModule::Get().CreateRequest();

HttpRequest->OnProcessRequestComplete().BindUObject(this, &UAsyncTaskDownloadImage::HandleImageRequest);
HttpRequest->SetURL(URL);
HttpRequest->SetVerb(TEXT("GET"));
HttpRequest->ProcessRequest();

I wrote my own download progress bar using

//IHttp
/**
 * Delegate called to update the request/response progress. See FHttpRequestProgressDelegate
 */
virtual FHttpRequestProgressDelegate& OnRequestProgress() = 0;

So I know the download is working, it just takes a really really long time compared to Chrome or my FTP server.

Thank you for any insights in this matter!

#A Neutral Test ~ 1GB Speed Test

If you want to test you can test with a 1 GB download here:

ftp://speedtest:speedtest@ftp.otenet.gr/test1Gb.db

I endured it, in UE4 it took:

17 minutes and 43 seconds

Exact same url, using python:

1 minute and 19 seconds

#My Conclusion

Clearly libcurl is throttled somewhere, and cannot use the full power of my internet connection, in a way that Chrome, my ftp server, and python are not limited.

Can you please address this so I can use UE4 instead of python to download DLC?

Thank youuuu!

:heart:

PS: Here is the python script I used:

import urllib2

downloadedfile = urllib2.urlopen("http://YourVeryLargeFile.exe")
with open('pO.exe', 'wb') as output:
    while True:
        data = downloadedfile.read(100000000)
        if data:
            output.write(data)
            print "read 100 mb"
        else:
            print "download complete and file written!"
            break

Sorry for being pedantic, but is it 15 or 20 times slower? The title and the body are conflicting.

UE4 http get is 15 x slower than python, and 20x slower than Chrome/my ftp server.

Also please see my neutral 1gb speed test which I just posted:

UE4 17 minutes 43 seconds

python 1 minute 19 seconds

I performed these test 1 right after the other, and not concurrently, same url :slight_smile:

Chrome, my ftp server, and python are all very similar, and UE4 http get using libcurl is at least 15x slower at a minimum.

1 Like

Might be the buffering, in your python script you just read 100000000 bytes while Epics code is buffering to throttle the download.

Hey ,

Thanks for the feedback. I was able to reproduce the issue and it is submitted as, UE-33732.

Thank you for the update!

Hi Moss!

Very nice to hear from you!

#:heart:

An important note for this issue. On windows we can switch to WinInet using the following code in defaultengine.ini:

[Networking]
 UseLibCurl=False

The problem is that Linux only has LibCurl and its way too slow. WinInet did work for my purposes though and is very fast. Another note: i tested LibCurl using the command line of LibCurl and all my calls were very fast as well, so i suspect it has something to do on how UE4 is using libcurl.

I figured out i actually had one special call in which i had a get request with an empty json as input (using varest plugin), apparently that caused the long http calls. It is somehow related to libcurl…

Is there any temporary solution for Android and IOS to get normal speed before this gets fixed?

There hasn’t been any update on a current work around or fix for this issue but another user has suggested something that may or may not work. You can see that here:

https://answers.unrealengine.com/questions/320627/varest-http-get-slow.html

For Android and IOS there is currently no workaround. The suggested workaround only works on windows, since there is support for WinInet there, which actually works fast. I am still suprised this is currently backed logged, how can you possibly be using HTTP with this speed? I consider it completely broken at the moment.

The only note on it is that it isn’t affecting anything bad enough to be critical. There is a plan to look at it whenever they can get to it, though.

This is already marked as resolved question, but I want to share I discovered. I found that HTTP request’s fetch rate is controlled by FHttpModule, especially HttpThreadActiveFrameTimeInSeconds variable is the key. Default value of it is around 0.005(200Hz). That means, tiny fetch operations were called just 200 times per second. By decreasing the value to the proper value I could get reasonable download speed.

You can control this value from the Engine configuration file like this:

[HTTP]
HttpThreadActiveFrameTimeInSeconds=0.00001
1 Like

Along the lines of nullbus’s answer, you can increase the buffer size to the libcurl maximum allowed value. For a large download over a very strong internet connection, this dropped my response times from ~8.3 seconds to ~1.3 seconds.

[HTTP.Curl]
BufferSize=524288

Thanks, this help for me.

Thanks, this works for us too.

About HTTP, maybe get some inspiration from this plugin: