Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating to v2.9.20 breaks cloudflared tunnel #2696

Closed
tech1ndex opened this issue Mar 17, 2023 · 61 comments
Closed

Updating to v2.9.20 breaks cloudflared tunnel #2696

tech1ndex opened this issue Mar 17, 2023 · 61 comments

Comments

@tech1ndex
Copy link

tech1ndex commented Mar 17, 2023

Environment

  • Docker running on Unraid Version: 6.11.1
  • cloudflare/cloudflared:2023.3.1
  • jc21/nginx-proxy-manager:2.9.20

Issue

  • After updating the Nginx-Proxy-Manager container to v.2.9.20 the following message is observed in cloudflared container logs:
ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: unrecognized name"
  • As a result , URLs behind Nginx-Proxy-Manager will not load

Mitigation

  • Downgrading back to v2.9.19 resolves the issue
@Hacksawfred3232
Copy link

Hacksawfred3232 commented Mar 17, 2023

You might be suffering the same issue I was having earlier, where an option being added - "ssl_reject_handshake" set to on - caused Nginx to reject SSL handshakes to the default handler. Unfortuantly, that blocks services that connects first, then uses the Host header to define where to send their requests. From testing, Apache2 defines who to connect to using SNI and that doesn't get blocked. Since SNI forwarding from NGINX to NPM appears to be broken at the moment, the fix is sticking to 2.9.19 if using NGINX or a service such as cloudflare.

The commit that introduces this problem: a7f0c3b

@TheBeeZee What was the rationale behind adding that option to the default config? Couldn't having that behind a togglable option been better, so frontends like NGINX standalone or cloudflare don't break?

@TheBeeZee
Copy link
Contributor

ssl_reject_handshake would only apply to incoming SSL requests without SNI.

The reporter of this issue doesn’t really provide enough information to be able to draw any conclusions.

tech1index, is Cloudflare configured to connect to NPM via https? If so, you would have to change the Cloudflare config to specify the host name it should connect to. You can do that in the public host name of your tunnel under Additional application settings -> TLS -> Origin server name.

@Hacksawfred3232
Copy link

Hacksawfred3232 commented Mar 17, 2023

ssl_reject_handshake would only apply to incoming SSL requests without SNI.
tech1index, is Cloudflare configured to connect to NPM via https?

"remote error: tls: unrecognized name" indicates that yes, Cloudflare(d) was connecting to NPM over HTTPS, since TLS is part of HTTPS.

While ssl_reject_handshake is useful, it also implies that NGINX knows what hostname it is asking for. Which is usually defined in a proxy_pass or server_name directive. Unless you start asking people to make explicit configurations for each and every hostname on both the NGINX frontend and the NPM backend, most configurations will have a simple "If it doesn't match anything else, forward to this backend" block that sends the request to NPM. If the frontend is Apache2, all is well and good, since Apache2 uses SNI in it's request. However, servers like NGINX will not use SNI in it's request to upstream, and that causes the request to be rejected.

Like I previously said, a global config option for ssl_reject_handshake instead of setting to on hardcoded would have been better, and will not break people's configurations when using frontend servers like NGINX.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 17, 2023 via email

@Hacksawfred3232
Copy link

NGINX supports SNI when proxying with proxy_ssl_server_name set to on: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_server_name

Tried that already, didn't work. Otherwise, would have mentioned it.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 17, 2023 via email

@tech1ndex
Copy link
Author

Okay tried the following:

  1. Created an Origin Cert as per the docs
  2. Added Original Cert and Key to NPM
  3. Added Original Cert to Host under Hosts > Proxy Hosts > myurl.com > SSL
  4. Added the following options under Hosts > Proxy Hosts > myurl.com > Edit > Advanced
proxy_ssl_server_name on;
proxy_ssl_name myurl.com;
  1. Restarted NPM, Restarted Cloudflared

Works in 2.9.19, but see the same error persisting in 2.9.20.

@Hacksawfred3232
Copy link

Hacksawfred3232 commented Mar 17, 2023

Depending on the rest of the config, you.might also have to specify the name nginx should be using with proxy_ssl_name.

Okay, tried the following:

proxy_ssl_server_name on;
proxy_ssl_name $host;

And that fixed the problem on 2.9.20. Will keep a patched version of 2.9.20 in case this issues arises again though. @tech1ndex Try using $host instead of myurl.com

@tech1ndex
Copy link
Author

@Hacksawfred3232

Thanks but unfortunately still not working for me on 2.9.20

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 17, 2023 via email

@tech1ndex
Copy link
Author

@TheBeeZee

Sure thing, NPM, Cloudflared and my target host are all running on the same Docker host in the same "proxynet" docker network I have created.

Host Application:

  • Web server listening on host port 5055

NPM Config is as follows:

  • Listens on host port 1433 (mapped to port 443) in container
  • Proxy Host config points to http://IP_of_Docker_Host:5055 with following options:

image

SSL Settings:

image

Advanced:

image

Cloudflared Config:

image

Please let me know if there is any additional info you need.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 17, 2023 via email

@Proximus888
Copy link

Proximus888 commented Mar 17, 2023

Having the same issue. I get a 502 error from cloudflared when I try to access my website.

NPM v2.9.19 everything works without a problem, NPM v2.9.20 gives me 502 errors. TLSverify disabled on clouflared tunnel.

I am running NPM on a Pi4 and running the services (Nextcloud, Bitwarden, etc..) on a NUC. Adding "proxy_ssl_server_name on;, proxy_ssl_name $host; " doesnt't resolve my issue.

In advanced tab

#Cloudflare IP and Basic Security
real_ip_header CF-Connecting-IP;
proxy_ssl_server_name on;
proxy_ssl_name $host;

For the moment reverting back to NPM v2.9.19.

@TheBeeZee adding the hostname to originServerName worked. Thanks for the info

@tech1ndex
Copy link
Author

I think you should be able to specify "originServerName: *.tech1index.ca" and remove "noTLSVerify: true", but I've never tried doing this with a wildcard certificate. If that doesn't work, you'd have to replace the wildcard with individual host names.

Right! That makes sense, and yes it won't work with a wildcard host but works when you break it out into individual hosts.

Traffic will still be encrypted between your clients and the Cloudflare frontend and then across the cloudflare tunnel, but will then be sent plaintext from cloudflared to NPM. This in no way jeopardizes the security of your setup, as the cloudflared-NPM connection is within your docker environment.

You are correct, when I had initially set this up I had CF and NPM on separate subnets and therefore wanted to keep the traffic encrypted across local networks however with the way it is setup now, it shouldn't be necessary at all. Thanks for pointing that out and for all your help!

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 17, 2023 via email

@marvagabi
Copy link

  1. Disable Force SSL and HTTP/2 support in NPM for your proxy host. 2. Instead of https on port 1443, you would forward to http on port 8080 (or whatever) in your cloudflared configuration.

Was having the same issue. These steps fixed it for me.

Changed Cloudflared config to not use https.
In NPM disabled "Force SSL" and "Http/2 Support"

Restarted the cloudflared docker and everything started working again.

@blaine07
Copy link

I don't fully understand all this as I am adjusting Cloudflared IN Cloudflare ONE dash board. I know TLSVerify has to be disabled. I Rolled back to .19 but moving forward what is going to happen here? Will we not be able to SSL all the way or is something going to be adjusted back with NPM or?

What's the real solution here?

@Proximus888
Copy link

I don't fully understand all this as I am adjusting Cloudflared IN Cloudflare ONE dash board. I know TLSVerify has to be disabled. I Rolled back to .19 but moving forward what is going to happen here? Will we not be able to SSL all the way or is something going to be adjusted back with NPM or?

What's the real solution here?

You can adjust these settings on the Cloudflare dashboard.

Go to your tunnel > public hostname > your website.

Then under 'additional application settings > tls, you will see 'Origin Server Name'. Just put there the hostname of your website. Like example.com or sub.example.com.

Then everything should work with ssl.

@blaine07
Copy link

blaine07 commented Mar 18, 2023

I don't fully understand all this as I am adjusting Cloudflared IN Cloudflare ONE dash board. I know TLSVerify has to be disabled. I Rolled back to .19 but moving forward what is going to happen here? Will we not be able to SSL all the way or is something going to be adjusted back with NPM or?
Screen Shot 2023-03-18 at 4 37 23 PM

What's the real solution here?

You can adjust these settings on the Cloudflare dashboard.

Go to your tunnel > public hostname > your website.

Then under 'additional application settings > tls, you will see 'Origin Server Name'. Just put there the hostname of your website. Like example.com or sub.example.com.

Then everything should work with ssl.

Yeah, I have had that there, but the one specifically giving me problems was a wildcard. Can this not be done with a wildcard? *.mydomain.net did NOT work.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 19, 2023 via email

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 19, 2023 via email

@blaine07
Copy link

blaine07 commented Mar 19, 2023

A wildcard won't work as nginx won't know which site you actually want to connect to. The hostname is sent as part of the TLS exchange, before any headers are exchanged, so Cloudflare in your example would need to send the actual hostname to NPM for NPM to know where to route the request to.

How I have above 100% works on 2.9.19; it quits on 2.9.20+. It forwards any subdomains I forwards to Cloudflared inside CF with the long dns address that points to that instance of Cloudflared.

My above pic, combined with this pic - the wildcard thing works fine:
Screen Shot 2023-03-18 at 9 07 02 PM

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 19, 2023 via email

@blaine07
Copy link

blaine07 commented Mar 19, 2023

On 2.9.19 it works by coincidence, not by design. With 2.9.19 the default SSL host is just a normal host, which uses an SSL certificate for a fake domain (localhost.localdomain or some such). If you connect to your NPM host using an invalid name or the IP address, it will present this dummy SSL certificate. By default cloudflare doesn't specify any hostname, so it lands on this default host. That's why you had to set "NoVerifyTLS" in the cloudflare configuration, which basically makes the entire connection insecure, completely defeating the purpose of TLS. With 2.9.20 the configuration was changed so that the default SSL host simply aborts the SSL negotiation if a hostname is not specified during TLS negotiation.

So moving forward, to fix, in cloudflare tunnel I need to remove wildcards and send EACH subdomain in individually? But doing them all individually I would or wouldn’t want to disable the noTLSVerify? Would I still set origin server name like in my first pic a few posts above this one?

Also, why was this change made in NPM. Was this by design and this change is sticking? The wild card subdomain thing simplified things a ton… I have a poop ton of subdomains I individually will have to setup.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 19, 2023 via email

@blaine07
Copy link

Correct, if you specify each subdomain individually it will work and no need to set noTLSverify. The second solution I suggested above was to set the origin server name to any of your subdomain (it doesn't matter which one) on the wildcard hostname tunnel. I don't have a setup where I could quickly verify that this works, but I don't see why it shouldn't.

I can’t make potentially breaking changes tonight, but could potentially play with it tomorrow. It makes sense now; I get it. I just don’t understand why change was necessary. Really probably need to just go through cloudflared and set each of my subdomains up individually but…

Sincerely, thank you, thank you so much for your patience in ensuring I understand. Thank you! I truly hope it helps this helps others who will soon start stumbling upon this, too. Thank YOU!

@blaine07
Copy link

Correct, if you specify each subdomain individually it will work and no need to set noTLSverify. The second solution I suggested above was to set the origin server name to any of your subdomain (it doesn't matter which one) on the wildcard hostname tunnel. I don't have a setup where I could quickly verify that this works, but I don't see why it shouldn't.

Also, doing them all individually would I still set origin name server too? I’d leave the noTLSVerifoy off but what about original name?

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 19, 2023 via email

@blaine07
Copy link

You would have to set the origin name on each tunnel host individually, correct. This ensures that CF will set SNI during TLS negotiation. If possible, please test my second suggestion (setting origin name to any of your subdomains on the wildcard), as that would be the quickest and simplest solution. Please report back here if it works or not.

Earlier I had a ton of issues with .21. Need to make sure my issues were only this and not the database issues discussed with .21 too. I think it was just this the way it was acting and now that I understand. But I did keep getting 503 errors trying to just get to NPM internally too though.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 20, 2023 via email

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Mar 24, 2023 via email

@yizhen22
Copy link

yizhen22 commented Apr 2, 2023

在 2.9.19 上,它的工作是巧合,而不是设计。对于 2.9.19,默认的 SSL 主机只是一个普通主机,它使用 SSL 证书作为假域(localhost.localdomain 或类似的)。如果您使用无效名称或 IP 地址连接到您的 NPM 主机,它将显示此虚拟 SSL 证书。默认情况下,cloudflare 不指定任何主机名,因此它位于此默认主机上。这就是为什么你必须在 cloudflare 配置中设置“NoVerifyTLS”,这基本上使整个连接不安全,完全违背了 TLS 的目的。在 2.9.20 中,配置已更改,如果在 TLS 协商期间未指定主机名,默认 SSL 主机将简单地中止 SSL 协商。
……
On Sat, Mar 18, 2023, at 7:03 PM, blaine07 wrote: > > A wildcard won't work as nginx won't know which site you actually want to connect to. The hostname is sent as part of the TLS exchange, before any headers are exchanged, so Cloudflare in your example would need to send the actual hostname to NPM for NPM to know where to route the request to. > … https://app.fastmail.com/mail/Inbox/compose/M38892bf5379b568f7e3f79c2?u=b1c14013# > How I have above 100% works on 2.9.19; it quits on 2.9.20+. It forwards any subdomains I forwards to Cloudflared inside CF with the long dns address that points to that instance of Cloudflared. — Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R44TYAVUCAXMJOHZYLW4ZSN7ANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

I also encountered this problem when using v2.10.2. This unexpected workflow may violate the security of TLS, but TLS only protects the corresponding domain name. If we use the domain name to access the Cloudflared tunnel through reverse proxy, I do not think that this workflow is distorted. It provides the possibility of intermediate webpage access, which can be useful in many scenarios. Even if website security is compromised and a third party may modify it, it is a compromise made out of helplessness. Adjusting it in the Cloudflared panel would make the application scenario more complicated.

v2.10.2 My approach is

cd yournginx/data/nginx/proxy_host
vi your.conf
  location / {
    proxy_ssl_server_name on;
    proxy_ssl_name $host;
    proxy_pass https://your;
    # Proxy!
    #include conf.d/include/proxy.conf;//To comment out this line
  }

cloudflared tunnel is back to normal

@crypticviper
Copy link

Correct, if you specify each subdomain individually it will work and no need to set noTLSverify. The second solution I suggested above was to set the origin server name to any of your subdomain (it doesn't matter which one) on the wildcard hostname tunnel. I don't have a setup where I could quickly verify that this works, but I don't see why it shouldn't.

Using a random subdomain did the trick for me. thanks a lot. Here is how my tunnel config looks like for reference.

ingress:
   - hostname: "*.mydomain.com"
     service: https://NGINXPROXYHOSTIP:443
     originRequest:
       originServerName: "anysubdomain.mydomain.com"
       noTLSVerify: true

   - service: http_status:404

@TheBeeZee
Copy link
Contributor

TheBeeZee commented May 1, 2023 via email

@boomam
Copy link

boomam commented Jun 22, 2023

Using a random subdomain did the trick for me. thanks a lot. Here is how my tunnel config looks like for reference.

ingress:
   - hostname: "*.mydomain.com"
     service: https://NGINXPROXYHOSTIP:443
     originRequest:
       originServerName: "anysubdomain.mydomain.com"
       noTLSVerify: true

   - service: http_status:404

...after a long day of dealing with a storm of outages and realizing that NPM's latest release does this TLS error with CF, this minor, super-small, but very effective tip on the originServerName as a resolution step, is the thing that is gonna let me go to bed at a decent time.
Thank you!

@andrewmooreio
Copy link

Sorry to comment on a closed issue, but if anyone could provide some advice on getting this working, it'd be greatly appreciated. I'm still getting the Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF 502 error from Cloudflare.

I've read through the thread and have implemented the fixes as follows:

  • Added "Origin Server Name" to Cloudflare tunnel config.
  • Added proxy_ssl_server_name on and proxy_ssl_name $host to custom Nginx config.

Am I missing something?

Screenshot 2023-07-07 at 17 42 15
Screenshot 2023-07-07 at 17 42 07
Screenshot 2023-07-07 at 17 41 31
Screenshot 2023-07-07 at 17 38 08

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 7, 2023 via email

@andrewmooreio
Copy link

andrewmooreio commented Jul 7, 2023

It appears that you are using a Cloudflare generated SSL certificate. Are you sure that certificate is for the hostname "overseerr*"? Also, try to temporary disable the access list in NPM to confirm that it is not incorrectly blocking incoming requests from Cloudflare. Additionally, while those are not the cause of your problem, you should probably still clean it up for clarity. First of all, the proxy_ssl_server_name and proxy_ssl_name is not needed in your setup, as NPM is not proxying to a http host. Also, you should disable "No TLS Verify" in the Cloudflare tunnel configuration - if everything is setup correctly it should not be needed and enabling it decreases the security of your setup.

On Fri, Jul 7, 2023, at 9:45 AM, Andrew Moore wrote: Sorry to comment on a closed issue, but if anyone could provide some advice on getting this working, it'd be greatly appreciated. I'm still getting the Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF 502 error from Cloudflare. I've read through the thread and have implemented the fixes as follows: • Added "Origin Server Name" to Cloudflare tunnel config. • Added proxy_ssl_server_name on and proxy_ssl_name $host to custom Nginx config. Am I missing something? Screenshot 2023-07-07 at 17 42 15 https://user-images.githubusercontent.com/20435317/251805092-37cc9d7e-04bf-4cc7-b6ea-73a3bd64397b.png Screenshot 2023-07-07 at 17 42 07 https://user-images.githubusercontent.com/20435317/251805095-2e2d3f9d-e8ad-4ee6-841c-deb9a24a80b3.png Screenshot 2023-07-07 at 17 41 31 https://user-images.githubusercontent.com/20435317/251805097-f5c1cab3-4856-4e66-a46f-04d2bae468f0.png Screenshot 2023-07-07 at 17 38 08 https://user-images.githubusercontent.com/20435317/251805102-5a5dea42-34e9-436b-9bbe-c8166b7d7397.png — Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R4HQPJ63RIEKFKI2WDXPA4MBANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

Thanks. The certificate I'm using is just a wildcard origin cert generated through the Cloudflare SSL/TLS dashboard. Covering *.domain.com and domain.com. Which I'm assuming is the right type? I've double-checked and both domains are included in the SAN.

I've disabled the ACL in NPM and the issue persists.

I'm keeping "No TLS Verify” enabled for now, just to make it easier. It should work without though, I agree.

Edit: I've just tried using a Let's Encrypt certificate on the origin and enabling "No TLS Verify" in Cloudflare. Same problem happens. It's strange that I'm having so much difficulty with this. If I point the Cloudflare tunnel at an origin server with a self-signed certificate I don't have an issue.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 7, 2023 via email

@andrewmooreio
Copy link

andrewmooreio commented Jul 7, 2023

Check the NPM logs. What do you see in the logs when you attempt to connect to your web site?

On Fri, Jul 7, 2023, at 11:57 AM, Andrew Moore wrote: > > It appears that you are using a Cloudflare generated SSL certificate. Are you sure that certificate is for the hostname "overseerr*"? Also, try to temporary disable the access list in NPM to confirm that it is not incorrectly blocking incoming requests from Cloudflare. Additionally, while those are not the cause of your problem, you should probably still clean it up for clarity. First of all, the proxy_ssl_server_name and proxy_ssl_name is not needed in your setup, as NPM is not proxying to a http host. Also, you should disable "No TLS Verify" in the Cloudflare tunnel configuration - if everything is setup correctly it should not be needed and enabling it decreases the security of your setup. > … https://app.fastmail.com/mail/Inbox/compose?u=b1c14013# > On Fri, Jul 7, 2023, at 9:45 AM, Andrew Moore wrote: Sorry to comment on a closed issue, but if anyone could provide some advice on getting this working, it'd be greatly appreciated. I'm still getting the Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF 502 error from Cloudflare. I've read through the thread and have implemented the fixes as follows: • Added "Origin Server Name" to Cloudflare tunnel config. • Added proxy_ssl_server_name on and proxy_ssl_name $host to custom Nginx config. Am I missing something? Screenshot 2023-07-07 at 17 42 15 https://user-images.githubusercontent.com/20435317/251805092-37cc9d7e-04bf-4cc7-b6ea-73a3bd64397b.png Screenshot 2023-07-07 at 17 42 07 https://user-images.githubusercontent.com/20435317/251805095-2e2d3f9d-e8ad-4ee6-841c-deb9a24a80b3.png Screenshot 2023-07-07 at 17 41 31 https://user-images.githubusercontent.com/20435317/251805097-f5c1cab3-4856-4e66-a46f-04d2bae468f0.png Screenshot 2023-07-07 at 17 38 08 https://user-images.githubusercontent.com/20435317/251805102-5a5dea42-34e9-436b-9bbe-c8166b7d7397.png — Reply to this email directly, view it on GitHub <#2696 (comment) <#2696 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R4HQPJ63RIEKFKI2WDXPA4MBANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.*> > Thanks. The certificate I'm using is just a wildcard origin cert generated through the Cloudflare SSL/TLS dashboard. Covering .domain.com and domain.com. Which I'm assuming is the right type? I've double-checked and both domains are included in the SAN. I've disabled the ACL in NPM and the issue persists. I'm keeping "No TLS Verify” enabled for now, just to make it easier. It should work without though, I agree. — Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R2T6FKVHIKGTVWYOALXPBLZTANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.>

Nothing at all! Which is strange. When connecting to the external domain (via Cloudflare) I see the following cloudflared errors:

2023-07-07T20:08:33Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" cfRay=7e32b857fa83777a-LHR event=1 ingressRule=3 originService=https://overseerr.domain.com:443
2023-07-07T20:08:33Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" connIndex=1 dest=https://request.domain.com/favicon.ico event=0 ip=198.41.192.77 type=http

However, proxy-host-15_access.log and proxy-host-15-error.log show nothing at all when accessing externally. When connecting locally, I do see access logs.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 7, 2023 via email

@andrewmooreio
Copy link

The name overseerr.andrewmoore.io doesn't seem to resolve for me. Are you sure DNS is setup correctly? Cloudflare obviously needs to be able to translate that name into an IP before it can connect to your origin server.

That'll only resolve locally, as it's the backend address. It resolves fine within the cloudflared container.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 7, 2023 via email

@andrewmooreio
Copy link

Nothing in any of the default log files.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 7, 2023 via email

@andrewmooreio
Copy link

andrewmooreio commented Jul 7, 2023

I can see the cloudflared container (172.17.0.4) making a DNS request for overseerr.andrewmoore.io (172.16.1.33) and then there is a key exchange with NPM (172.16.1.33), no traffic at all after that key exchange.

Screenshot 2023-07-07 at 22 09 06

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 7, 2023 via email

@andrewmooreio
Copy link

Please see below for the output:

depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify return:1
DONE
                DNS:*.andrewmoore.io, DNS:andrewmoore.io

@andrewmooreio
Copy link

I'm not sure if the Cloudflare certificate is the issue. If I use a Let's Encrypt certificate, which I know works because my browser trusts it, and enable “No TLS Verify” on Cloudflare, the issue still persists.

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 8, 2023 via email

@andrewmooreio
Copy link

I think we're moving in the right direction to finding a resolution, but it's not making a lot of sense to me.

Changed the NPM proxy host to HTTP only, with the Cloudflare tunnel configuration to match, hitting http://overseerr.andrewmoore.io:80

Going to https://request.andrewmoore.io now returns a 404 from openresty, which I guess is a good thing as we're now at least hitting the web server. default-host_access.log shows:
172.17.0.1 - - [08/Jul/2023:15:11:34 +0000] "GET / HTTP/1.1" 404 183 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"

cloudflared log:
2023-07-08T15:14:05Z DBG GET https://request.andrewmoore.io/ HTTP/1.1 cfRay=7e39465e2dae71ed-LHR connIndex=0 content-length=0 event=1 headers={"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Encoding":["gzip"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Cache-Control":["max-age=0"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["82.71.18.148"],"Cf-Ipcountry":["GB"],"Cf-Ray":["7e39465e2dae71ed-LHR"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["079b8679-7ae3-4f7c-88fd-1dc6b7be020b"],"Cookie":["__cf_bm=wPxweKQxRSBleQcIGufRvojriwwyJf4iO2_9ONFT2vo-1688828570-0-AaO+PNy2JZS1aed6aORjtA1VNeVSmURXSFqb+A89ZqKixd5Aqf5Ll+eBoG2GdhwR5w=="],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Google Chrome\";v=\"114\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"],"X-Forwarded-For":["82.71.18.148"],"X-Forwarded-Proto":["https"]} host=request.andrewmoore.io ingressRule=3 path=/ 2023-07-08T15:14:05Z DBG 404 Not Found cfRay=7e39465e2dae71ed-LHR connIndex=0 content-length=-1 event=1

Hitting the same endpoint from anywhere locally works without issue.

# curl -IL http://overseerr.andrewmoore.io:80
HTTP/1.1 307 Temporary Redirect
Server: openresty
Date: Sat, 08 Jul 2023 15:16:39 GMT
Connection: keep-alive
X-Powered-By: Express
Location: /login
X-Served-By: overseerr.andrewmoore.io

HTTP/1.1 200 OK
Server: openresty
Date: Sat, 08 Jul 2023 15:16:39 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
X-Powered-By: Next.js
Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate
Vary: Accept-Encoding
X-Served-By: overseerr.andrewmoore.io

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 8, 2023 via email

@andrewmooreio
Copy link

Gotcha, that's now working with the additional HTTP Host Header.

@andrewmooreio
Copy link

That was it all along. HTTPS is now working too with the HTTP Host Header configured.

Thanks so much for walking me through this. Sorry the issue wasn't more interesting haha!

@TheBeeZee
Copy link
Contributor

TheBeeZee commented Jul 8, 2023 via email

@AvatarMomo
Copy link

AvatarMomo commented Jul 14, 2023

I was getting this for v2.10.3. I fixed this by updating the cloudflared config. Here is how it looks now. Previously I was using a wildcard rule to catch all subdomains but this no longer works.

credentials-file: /home/nonroot/.cloudflared/xxxxxxxx.json
loglevel: info
ingress:
  - hostname: domain.tld
    service: https://localhost:port
    originRequest:
      noTLSVerify: true
      originServerName: domain.tld
      httpHostHeader: domain.tld
  - hostname: subd.domain.tld
    service: https://localhost:port
    originRequest:
      noTLSVerify: true
      originServerName: subd.domain.tld
      httpHostHeader: subd.domain.tld
   - service: http_status:404

I did not change anything in the npm advanced setting for each proxy host.

@m1xxos
Copy link

m1xxos commented Apr 17, 2024

I was getting this for v2.10.3. I fixed this by updating the cloudflared config. Here is how it looks now. Previously I was using a wildcard rule to catch all subdomains but this no longer works.

credentials-file: /home/nonroot/.cloudflared/xxxxxxxx.json
loglevel: info
ingress:
  - hostname: domain.tld
    service: https://localhost:port
    originRequest:
      noTLSVerify: true
      originServerName: domain.tld
      httpHostHeader: domain.tld
  - hostname: subd.domain.tld
    service: https://localhost:port
    originRequest:
      noTLSVerify: true
      originServerName: subd.domain.tld
      httpHostHeader: subd.domain.tld
   - service: http_status:404

I did not change anything in the npm advanced setting for each proxy host.

Thank you, this configuration worked for me on NPM 2.11.1 version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests