Caching steam and updates for fun and profit(?)

[Tom]

Based on http://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/

I have set up a similar setup to the above blog, but with some changes due to their setup not being "complete" in that they didn't publish their custom nginx module, and that they didn't document everything.

I started with a (slightly customized using debian's support for automated installs, can post about this later too if anyone cares ) bare debian install in a vm, then installed nginx. As I already have a local bind instance set up as a caching name server for my lan, I was able to extend it to support this use case by configuring a Response Policy Zone (see: http://jpmens.net/2011/04/26/how-to-configure-your-bind-resolvers-to-lie-using-response-policy-zones-rpz/) to lie about a few domains (as many steam content zones, debian mirrors, and microsoft/xbox servers as I could, forwarding xbox just because it was part of the blog's config) to forward them to the lancache vm's nginx install.

You'll notice separate host names per cache type, this just makes it easier to set up nginx virtual hosts, each host has its own static ip. I haven't checked, but individual IPs may not be required, unless various clients don't support name based virtual hosts.

Code Select Expand

; rpz
$TTL 600
@ IN     SOA    firewall.home.tomasu.org. thomas@fjellstrom.ca. (
                                2016101601  ; Serial
                                3H          ; refresh after 3 hours
                                1H          ; retry after 1 hour
                                1W          ; expire after 1 week
                                1D)         ; minimum TTL of 1 day

@                                       IN NS firewall.home.tomasu.org.

; STEAM
cs.steampowered.com                     CNAME lancache-steam.svc.tomasu.org.
*.cs.steampowered.com                   CNAME lancache-steam.svc.tomasu.org.
content1.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
content2.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
content3.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
content4.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
content5.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
content6.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
content7.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
content8.steampowered.com               CNAME lancache-steam.svc.tomasu.org.
*.hsar.steampowered.com.edgesuite.net   CNAME lancache-steam.svc.tomasu.org.
*.akamai.steamstatic.com                CNAME lancache-steam.svc.tomasu.org.
content-origin.steampowered.com         CNAME lancache-steam.svc.tomasu.org.
client-download.steampowered.com        CNAME lancache-steam.svc.tomasu.org.
*.steamcontent.com                      CNAME lancache-steam.svc.tomasu.org.


; Microsoft

*.download.windowsupdate.com    CNAME lancache-microsoft.svc.tomasu.org.
download.windowsupdate.com      CNAME lancache-microsoft.svc.tomasu.org.
dlassets.xboxlive.com           CNAME lancache-microsoft.svc.tomasu.org.
*.xboxone.loris.llnwd.net       CNAME lancache-microsoft.svc.tomasu.org.
xboxone.vo.llnwd.net            CNAME lancache-microsoft.svc.tomasu.org.
images-eds.xboxlive.com         CNAME lancache-microsoft.svc.tomasu.org.
xbox-mbr.xboxlive.com           CNAME lancache-microsoft.svc.tomasu.org.
assets1.xboxlive.com.nsatc.net  CNAME lancache-microsoft.svc.tomasu.org.
assets1.xboxlive.com            CNAME lancache-microsoft.svc.tomasu.org.

; Debian

httpredir.debian.org            CNAME lancache-debian.svc.tomasu.org.
http.debian.net                 CNAME lancache-debian.svc.tomasu.org.
ftp.debian.org                  CNAME lancache-debian.svc.tomasu.org.
http.us.debian.org              CNAME lancache-debian.svc.tomasu.org.
http.ca.debian.org              CNAME lancache-debian.svc.tomasu.org.
ftp.us.debian.org               CNAME lancache-debian.svc.tomasu.org.
ftp.ca.debian.org               CNAME lancache-debian.svc.tomasu.org.
security.debian.org             CNAME lancache-debian.svc.tomasu.org.

ppa.launchpad.net               CNAME lancache-debian.svc.tomasu.org.

mirrors.kernel.org              CNAME lancache-debian.svc.tomasu.org.

; Google?! let it cache using debian config... some debian packages will arrive via there anyhow
; can be used via https which is harder to set up...
;dl.google.com                  CNAME lancache-debian.svc.tomasu.org.


The tricky part seems to be properly caching windows and xbox updates. Steam and Debian updates are simple and use normal HTTP downloads with optional Range fetches. Windows updates on the other hand will use semi random range's on different clients which can make caching responses nearly impossible. NGINX at one point added a feature to help with this called "slices". It basically takes incoming Range requests and un-mangles them to align on stable boundaries, which makes it possible to cache individual chunks of update files, while still allowing the (imo broken) windows and xbox updaters to randomize the requests.

Nginx config:

Code Select Expand

# Microsoft Node
include lancache/resolver;

# Xbox One downloader:
# * Uses a single archive file with range requests.
# * Uses redirects sometimes to push off to 3rd party CDNs.
# * Mainly uses 696320 byte requests but overlaps quite a bit
#   so we use double that.

# Ensure we don't cache redirects to 3rd party CDNs
error_page 301 302 307 = @redirect;
recursive_error_pages on;
proxy_intercept_errors on;

proxy_hide_header Etag;
proxy_cache_key "$server_name$request_uri $slice_range";
proxy_cache_valid 206 90d;
#range_cache_size 1392640;  <-- this is a command for the missing module, so we comment it out

location / {
        # proxy_set_header directives aren't merged so must be here

        slice 1m;
        proxy_set_header Range $slice_range;
        include lancache/cache-installs;
}

# Process a redirect and cache the response
location @redirect {
        slice 1m;
        # proxy_set_header directives aren't merged so must be here
        proxy_set_header Range $slice_range;
        include lancache/cache-installs-base;
        # We need to copy the variable as it's not available after the pass starts
        set $location $upstream_http_location;
        proxy_pass $location;
}


One problem I ran into is that the ftp.us.debian.org host just isn't working. So far any requests to it don't hit the nginx server for some odd reason. Not important, just curious. Another is that debian downloads through this setup seem slower than normal, not sure why. Steam is plenty fast. Also a slight issue with dl.google.com, it will forward regular http requests to https, which breaks with this setup as is. Some clever ssl spoofing will be needed there if it gets added.

I have tested the windows updates, and i get proxy cache hits, so It does seem to work reasonably well.

IMO this is easier to set up than Squid, and should be higher performance due to not hijacking all http/tcp requests. If you're worried about clients not using your dns and thus not hitting your dns spoofs, just setup your firewall to forward all dns traffic to your dns server, all except the lancache nginx server, it still needs to talk to an external, non spoofed dns server.

So there we go. Cool update caching for fun and profit.

[Lazybones]

FYI the use of http ranges is actually a performance thing. To get over TCP latency YouTube, WSUS and a few other segment downloads and send the requests to multiple hosts. It can result in much greater combined download speeds for a single large file. However it is a bitch to try and cache.   

[Tom]

The issue is more with the wu and xbox downloaders randomizing those ranges, rather than just using ranges. Steam uses predictible chunks and works more or less out of the box. Supporting xbox and wu downloads is a bit of a trick. The nginx slice module takes the incomming range requests and repacks them into alligned requests that it can cache, then it spits back out the requested data once it gets it. Makes it so it can cache the data, while still giving the client the data its expecting.

[Lazybones]

Caching YouTube for cheap would be handy. Every now and  again a corporate video gets uploaded and I have many users all jumping on at once to watch the same video.

I have seen some past scripts and paid addons for squid in the past.

[Tom]

I imagine this setup could do youtube. not really worth it for me though.