New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updating to v2.9.20 breaks cloudflared tunnel #2696
Comments
You might be suffering the same issue I was having earlier, where an option being added - "ssl_reject_handshake" set to on - caused Nginx to reject SSL handshakes to the default handler. Unfortuantly, that blocks services that connects first, then uses the Host header to define where to send their requests. From testing, Apache2 defines who to connect to using SNI and that doesn't get blocked. Since SNI forwarding from NGINX to NPM appears to be broken at the moment, the fix is sticking to 2.9.19 if using NGINX or a service such as cloudflare. The commit that introduces this problem: a7f0c3b @TheBeeZee What was the rationale behind adding that option to the default config? Couldn't having that behind a togglable option been better, so frontends like NGINX standalone or cloudflare don't break? |
ssl_reject_handshake would only apply to incoming SSL requests without SNI. The reporter of this issue doesn’t really provide enough information to be able to draw any conclusions. tech1index, is Cloudflare configured to connect to NPM via https? If so, you would have to change the Cloudflare config to specify the host name it should connect to. You can do that in the public host name of your tunnel under Additional application settings -> TLS -> Origin server name. |
"remote error: tls: unrecognized name" indicates that yes, Cloudflare(d) was connecting to NPM over HTTPS, since TLS is part of HTTPS. While ssl_reject_handshake is useful, it also implies that NGINX knows what hostname it is asking for. Which is usually defined in a proxy_pass or server_name directive. Unless you start asking people to make explicit configurations for each and every hostname on both the NGINX frontend and the NPM backend, most configurations will have a simple "If it doesn't match anything else, forward to this backend" block that sends the request to NPM. If the frontend is Apache2, all is well and good, since Apache2 uses SNI in it's request. However, servers like NGINX will not use SNI in it's request to upstream, and that causes the request to be rejected. Like I previously said, a global config option for ssl_reject_handshake instead of setting to on hardcoded would have been better, and will not break people's configurations when using frontend servers like NGINX. |
NGINX supports SNI when proxying with proxy_ssl_server_name set to on:
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_server_name
…On Thu, Mar 16, 2023, at 10:44 PM, Hacksawfred3232 wrote:
>
> ssl_reject_handshake would only apply to incoming SSL requests without SNI.
> tech1index, is Cloudflare configured to connect to NPM via https?
> "remote error: tls: unrecognized name" indicates that yes, Cloudflare(d) was connecting to NPM over HTTPS, since TLS is part of HTTPS.
>
While ssl_reject_handshake is useful, it also implies that NGINX knows *what* hostname it is asking for. Which is usually defined in a proxy_pass or server_name directive. Unless you start asking people to make explicit configurations for each and every hostname on both the NGINX frontend and the NPM backend, most configurations will have a simple "If it doesn't match anything else, forward to this backend" block that sends the request to NPM. If the frontend is Apache2, all is well and good, since Apache2 uses SNI in it's request. However, servers like NGINX will not use SNI in it's request to upstream, and that causes the request to be rejected.
Like I previously said, a global config option for ssl_reject_handshake instead of setting to on hardcoded would have been better, and will not break people's configurations when using frontend servers like NGINX.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5RZHVRDHHVRQOS7VFATW4P24RANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Tried that already, didn't work. Otherwise, would have mentioned it. |
Depending on the rest of the config, you.might also have to specify the name nginx should be using with proxy_ssl_name.
…On Thu, Mar 16, 2023, at 11:03 PM, Hacksawfred3232 wrote:
> NGINX supports SNI when proxying with proxy_ssl_server_name set to on: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_server_name
>
Tried that already, didn't work. Otherwise, would have mentioned it.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R6XH7UQ6NHOQMD47G3W4P5EXANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Okay tried the following:
Works in 2.9.19, but see the same error persisting in 2.9.20. |
Okay, tried the following:
And that fixed the problem on 2.9.20. Will keep a patched version of 2.9.20 in case this issues arises again though. @tech1ndex Try using |
Thanks but unfortunately still not working for me on 2.9.20 |
Victor, can you describe your setup?
…On Fri, Mar 17, 2023, at 9:41 AM, Victor Bajada wrote:
@Hacksawfred3232 <https://github.com/Hacksawfred3232>
Thanks but unfortunately still not working for me on 2.9.20
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5RZE2YMNFKQDKT6XJZDW4SH4XANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Sure thing, NPM, Cloudflared and my target host are all running on the same Docker host in the same "proxynet" docker network I have created. Host Application:
NPM Config is as follows:
SSL Settings: Advanced: Cloudflared Config: Please let me know if there is any additional info you need. |
Thanks, I see where the issue is. The problem is that cloudflared is not passing the host name it wants to connect to through SNI. One way to fix this would be to add a originServerName with the host name you want to connect to under "ingress" in your cloudflared configuration. Here's a link to the doc:
https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/local/local-management/ingress/#originservername
I think you should be able to specify "originServerName: *.tech1index.ca" and remove "noTLSVerify: true", but I've never tried doing this with a wildcard certificate. If that doesn't work, you'd have to replace the wildcard with individual host names.
I'm curious though, Is there a reason why you need to use https at all internally between cloudflared and NPM? It doesn't really buy you anything and it just wastes CPU because you are decrypting and re-encrypting your web traffic between cloudflared and NPM, which is all within your docker environment. Instead, you could:
1. Disable Force SSL and HTTP/2 support in NPM for your proxy host.
2. Instead of https on port 1443, you would forward to http on port 8080 (or whatever) in your cloudflared configuration.
Traffic will still be encrypted between your clients and the Cloudflare frontend and then across the cloudflare tunnel, but will then be sent plaintext from cloudflared to NPM. This in no way jeopardizes the security of your setup, as the cloudflared-NPM connection is within your docker environment.
…On Fri, Mar 17, 2023, at 9:57 AM, Victor Bajada wrote:
@TheBeeZee <https://github.com/TheBeeZee>
Sure thing, NPM, Cloudflared and my target host are all running on the same Docker host in the same "proxynet" docker network I have created.
Host Application:
• Web server listening on host port 5055
NPM Config is as follows:
• Listens on host port 1433 (mapped to port 443) in container
• Proxy Host config points to `http://IP_of_Docker_Host:5055` with following options:
image <https://user-images.githubusercontent.com/50547519/225968787-fa860f69-4a46-44bc-89fd-830decd65509.png>
SSL Settings:
image <https://user-images.githubusercontent.com/50547519/225968892-27111944-b8d9-4214-a2b4-5e4f75c3645d.png>
Advanced:
image <https://user-images.githubusercontent.com/50547519/225968968-1cbdba03-5d2d-492e-83ac-377a5b312b3a.png>
Cloudflared Config:
image <https://user-images.githubusercontent.com/50547519/225969634-9525e549-f24f-44e3-a773-79ef64132109.png>
Please let me know if there is any additional info you need.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R24ACYDSHTVBV7TFU3W4SJZHANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Having the same issue. I get a 502 error from cloudflared when I try to access my website. NPM v2.9.19 everything works without a problem, NPM v2.9.20 gives me 502 errors. TLSverify disabled on clouflared tunnel. I am running NPM on a Pi4 and running the services (Nextcloud, Bitwarden, etc..) on a NUC. Adding "proxy_ssl_server_name on;, proxy_ssl_name $host; " doesnt't resolve my issue. In advanced tab
For the moment reverting back to NPM v2.9.19. @TheBeeZee adding the hostname to originServerName worked. Thanks for the info |
Right! That makes sense, and yes it won't work with a wildcard host but works when you break it out into individual hosts.
You are correct, when I had initially set this up I had CF and NPM on separate subnets and therefore wanted to keep the traffic encrypted across local networks however with the way it is setup now, it shouldn't be necessary at all. Thanks for pointing that out and for all your help! |
You need to specify the hostname to connect to in the cloudflared configuration (see my response to Victor). Remove the proxy_ssl_server_name stuff from NPM, it is not needed in this case. (It would be needed if NPM was proxying *to* a host that needs SNI).
On Fri, Mar 17, 2023, at 12:16 PM, Proximus888 wrote:
Having the same issue. I get a 502 error from cloudflared when I try to access my website.
NPM v2.9.19 everything works without a problem, NPM v2.9.20 gives me 502 errors. TLSverify disabled on clouflared tunnel.
I am running NPM on a Pi4 and running the services (Nextcloud, Bitwarden, etc..) on a NUC. Adding "proxy_ssl_server_name on;, proxy_ssl_name $host; " doesnt't resolve my issue.
In advanced tab
`#Cloudflare IP and Basic Security
real_ip_header CF-Connecting-IP;
proxy_ssl_server_name on;
proxy_ssl_name $host;
`
… For the moment reverting back to NPM v2.9.19.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5RZ3QAEV2KKZKF3HDS3W4S2AHANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Was having the same issue. These steps fixed it for me. Changed Cloudflared config to not use https. Restarted the cloudflared docker and everything started working again. |
I don't fully understand all this as I am adjusting Cloudflared IN Cloudflare ONE dash board. I know TLSVerify has to be disabled. I Rolled back to .19 but moving forward what is going to happen here? Will we not be able to SSL all the way or is something going to be adjusted back with NPM or? What's the real solution here? |
You can adjust these settings on the Cloudflare dashboard. Go to your tunnel > public hostname > your website. Then under 'additional application settings > tls, you will see 'Origin Server Name'. Just put there the hostname of your website. Like example.com or sub.example.com. Then everything should work with ssl. |
A wildcard won't work as nginx won't know which site you actually want to connect to. The hostname is sent as part of the TLS exchange, before any headers are exchanged, so Cloudflare in your example would need to send the actual hostname to NPM for NPM to know where to route the request to.
…On Sat, Mar 18, 2023, at 2:35 PM, blaine07 wrote:
>>
>> I don't fully understand all this as I am adjusting Cloudflared IN Cloudflare ONE dash board. I know TLSVerify has to be disabled. I Rolled back to .19 but moving forward what is going to happen here? Will we not be able to SSL all the way or is something going to be adjusted back with NPM or?
>> What's the real solution here?
>>
> You can adjust these settings on the Cloudflare dashboard.
>
> Go to your tunnel > public hostname > your website.
>
> Then under 'additional application settings > tls, you will see 'Origin Server Name'. Just put there the hostname of your website. Like example.com or sub.example.com.
>
> Then everything should work with ssl.
>
Yeah, I have had that there, but the one specifically giving me problems was a wildcard. Can this not be done with a wildcard? *.mydomain.net did NOT work.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R6BBOKGGDTAKY72O6LW4YTA7ANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Actually, thinking about this again, you can probably make it work by specifying one of your hostnames (any, it doesn't matter which) in the Cloudflare configuration for the tunnel (tunnel > public hostname > your website, then under under 'additional application settings > tls, you will see 'Origin Server Name'). This will make the SSL negotiation succeed with SNI, then the request will be routed to the correct host using the Host header specified in HTTP.
…On Sat, Mar 18, 2023, at 6:39 PM, Blaž Zupan wrote:
A wildcard won't work as nginx won't know which site you actually want to connect to. The hostname is sent as part of the TLS exchange, before any headers are exchanged, so Cloudflare in your example would need to send the actual hostname to NPM for NPM to know where to route the request to.
On Sat, Mar 18, 2023, at 2:35 PM, blaine07 wrote:
>
>
>>>
>>>
>>> I don't fully understand all this as I am adjusting Cloudflared IN Cloudflare ONE dash board. I know TLSVerify has to be disabled. I Rolled back to .19 but moving forward what is going to happen here? Will we not be able to SSL all the way or is something going to be adjusted back with NPM or?
>>> What's the real solution here?
>>>
>>>
>> You can adjust these settings on the Cloudflare dashboard.
>>
>> Go to your tunnel > public hostname > your website.
>>
>> Then under 'additional application settings > tls, you will see 'Origin Server Name'. Just put there the hostname of your website. Like example.com or sub.example.com.
>>
>> Then everything should work with ssl.
>>
> Yeah, I have had that there, but the one specifically giving me problems was a wildcard. Can this not be done with a wildcard? *.mydomain.net did NOT work.
>
>
>
> —
> Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R6BBOKGGDTAKY72O6LW4YTA7ANCNFSM6AAAAAAV55G3CY>.
> You are receiving this because you were mentioned.Message ID: ***@***.***>
>
>
|
How I have above 100% works on 2.9.19; it quits on 2.9.20+. It forwards any subdomains I forwards to Cloudflared inside CF with the long dns address that points to that instance of Cloudflared. My above pic, combined with this pic - the wildcard thing works fine: |
On 2.9.19 it works by coincidence, not by design. With 2.9.19 the default SSL host is just a normal host, which uses an SSL certificate for a fake domain (localhost.localdomain or some such). If you connect to your NPM host using an invalid name or the IP address, it will present this dummy SSL certificate. By default cloudflare doesn't specify any hostname, so it lands on this default host. That's why you had to set "NoVerifyTLS" in the cloudflare configuration, which basically makes the entire connection insecure, completely defeating the purpose of TLS.
With 2.9.20 the configuration was changed so that the default SSL host simply aborts the SSL negotiation if a hostname is not specified during TLS negotiation.
…On Sat, Mar 18, 2023, at 7:03 PM, blaine07 wrote:
>
> A wildcard won't work as nginx won't know which site you actually want to connect to. The hostname is sent as part of the TLS exchange, before any headers are exchanged, so Cloudflare in your example would need to send the actual hostname to NPM for NPM to know where to route the request to.
> … <https://app.fastmail.com/mail/Inbox/compose/M38892bf5379b568f7e3f79c2?u=b1c14013#>
>
How I have above 100% works on 2.9.19; it quits on 2.9.20+. It forwards any subdomains I forwards to Cloudflared inside CF with the long dns address that points to that instance of Cloudflared.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R44TYAVUCAXMJOHZYLW4ZSN7ANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
So moving forward, to fix, in cloudflare tunnel I need to remove wildcards and send EACH subdomain in individually? But doing them all individually I would or wouldn’t want to disable the noTLSVerify? Would I still set origin server name like in my first pic a few posts above this one? Also, why was this change made in NPM. Was this by design and this change is sticking? The wild card subdomain thing simplified things a ton… I have a poop ton of subdomains I individually will have to setup. |
Correct, if you specify each subdomain individually it will work and no need to set noTLSverify.
The second solution I suggested above was to set the origin server name to *any* of your subdomain (it doesn't matter which one) on the wildcard hostname tunnel. I don't have a setup where I could quickly verify that this works, but I don't see why it shouldn't.
…On Sat, Mar 18, 2023, at 7:25 PM, blaine07 wrote:
>
> On 2.9.19 it works by coincidence, not by design. With 2.9.19 the default SSL host is just a normal host, which uses an SSL certificate for a fake domain (localhost.localdomain or some such). If you connect to your NPM host using an invalid name or the IP address, it will present this dummy SSL certificate. By default cloudflare doesn't specify any hostname, so it lands on this default host. That's why you had to set "NoVerifyTLS" in the cloudflare configuration, which basically makes the entire connection insecure, completely defeating the purpose of TLS. With 2.9.20 the configuration was changed so that the default SSL host simply aborts the SSL negotiation if a hostname is not specified during TLS negotiation.
> … <https://app.fastmail.com/mail/Inbox/compose/M9b82ca8e430a9158636e4752?u=b1c14013#>
>
So moving forward, to fix, in cloudflare tunnel in need to remove wildcard and send EACH subdomain in individually? But doing them all individually I wouldn’t need to disable the noTLSVerify? Would I still set origin server name like in my first pic a few posts above this one?
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R7RMYYIFXLFFWY7CGLW4ZVAVANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I can’t make potentially breaking changes tonight, but could potentially play with it tomorrow. It makes sense now; I get it. I just don’t understand why change was necessary. Really probably need to just go through cloudflared and set each of my subdomains up individually but… Sincerely, thank you, thank you so much for your patience in ensuring I understand. Thank you! I truly hope it helps this helps others who will soon start stumbling upon this, too. Thank YOU! |
Also, doing them all individually would I still set origin name server too? I’d leave the noTLSVerifoy off but what about original name? |
You would have to set the origin name on each tunnel host individually, correct. This ensures that CF will set SNI during TLS negotiation.
If possible, please test my second suggestion (setting origin name to any of your subdomains on the wildcard), as that would be the quickest and simplest solution. Please report back here if it works or not.
…On Sat, Mar 18, 2023, at 7:33 PM, blaine07 wrote:
>
> Correct, if you specify each subdomain individually it will work and no need to set noTLSverify. The second solution I suggested above was to set the origin server name to *any* of your subdomain (it doesn't matter which one) on the wildcard hostname tunnel. I don't have a setup where I could quickly verify that this works, but I don't see why it shouldn't.
> … <https://app.fastmail.com/mail/Inbox/compose/Ma0f14c80fa6d5c23bf79124c?u=b1c14013#>
>
Also, doing them all individually would I still set origin name server too? I’d leave the noTLSVerifoy off but what about original name?
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R4VVOUM4EVZ53DJFWDW4ZV7FANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Earlier I had a ton of issues with .21. Need to make sure my issues were only this and not the database issues discussed with .21 too. I think it was just this the way it was acting and now that I understand. But I did keep getting 503 errors trying to just get to NPM internally too though. |
Just a quick update here. I was able to setup a test to confirm that setting the origin server name in the TLS settings for the CF tunnel makes things work when the origin server is using TLS with a wildcard certificate.
…On Sat, Mar 18, 2023, at 7:39 PM, blaine07 wrote:
>
> You would have to set the origin name on each tunnel host individually, correct. This ensures that CF will set SNI during TLS negotiation. If possible, please test my second suggestion (setting origin name to any of your subdomains on the wildcard), as that would be the quickest and simplest solution. Please report back here if it works or not.
> … <https://app.fastmail.com/mail/Archive/compose/Mff8fda06a08fd987bbf52d98?u=b1c14013#>
>
Earlier I had a ton of issues with .21. Need to make sure my issues were only this and not the database issues discussed with .21 too. I think it was just this the way it was acting and now that I understand. But I did keep getting 503 errors trying to just get to NPM internally too though.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R3LYUZFD2P5GV4ZHYTW4ZWUZANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Sorry, I'm not familiar with haproxy. Do a Google search for "haproxy sni backend", that should come up with some solutions.
…On Fri, Mar 24, 2023, at 10:14 AM, msalman-91 wrote:
Hello @TheBeeZee <https://github.com/TheBeeZee>
i'm having the same issue but my npm is behind haproxy this blocks the first connection between haproxy and npm on port 443.
Do you have a way to add sni on haproxy.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R3TRSHC7BWRTAERMF3W5XJALANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I also encountered this problem when using v2.10.2. This unexpected workflow may violate the security of TLS, but TLS only protects the corresponding domain name. If we use the domain name to access the Cloudflared tunnel through reverse proxy, I do not think that this workflow is distorted. It provides the possibility of intermediate webpage access, which can be useful in many scenarios. Even if website security is compromised and a third party may modify it, it is a compromise made out of helplessness. Adjusting it in the Cloudflared panel would make the application scenario more complicated. v2.10.2 My approach is cd yournginx/data/nginx/proxy_host
vi your.conf location / {
proxy_ssl_server_name on;
proxy_ssl_name $host;
proxy_pass https://your;
# Proxy!
#include conf.d/include/proxy.conf;//To comment out this line
} cloudflared tunnel is back to normal |
Using a random subdomain did the trick for me. thanks a lot. Here is how my tunnel config looks like for reference. ingress:
- hostname: "*.mydomain.com"
service: https://NGINXPROXYHOSTIP:443
originRequest:
originServerName: "anysubdomain.mydomain.com"
noTLSVerify: true
- service: http_status:404
|
You can remove noTLSVerify.
…On Mon, May 1, 2023, at 2:56 AM, Viper wrote:
> Correct, if you specify each subdomain individually it will work and no need to set noTLSverify. The second solution I suggested above was to set the origin server name to *any* of your subdomain (it doesn't matter which one) on the wildcard hostname tunnel. I don't have a setup where I could quickly verify that this works, but I don't see why it shouldn't.
>
Using a random subdomain did the trick for me. thanks a lot. Here is how my tunnel config looks like for reference.
ingress:
- hostname: "*.mydomain.com"
service: https://NGINXPROXYHOSTIP:443
originRequest:
originServerName: "anysubdomain.mydomain.com"
noTLSVerify: true
- service: http_status:404
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5RYN7GFK2V6DPUEAUR3XD6CDNANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
...after a long day of dealing with a storm of outages and realizing that NPM's latest release does this TLS error with CF, this minor, super-small, but very effective tip on the originServerName as a resolution step, is the thing that is gonna let me go to bed at a decent time. |
Sorry to comment on a closed issue, but if anyone could provide some advice on getting this working, it'd be greatly appreciated. I'm still getting the I've read through the thread and have implemented the fixes as follows:
Am I missing something? |
It appears that you are using a Cloudflare generated SSL certificate. Are you sure that certificate is for the hostname "overseerr*"? Also, try to temporary disable the access list in NPM to confirm that it is not incorrectly blocking incoming requests from Cloudflare.
Additionally, while those are not the cause of your problem, you should probably still clean it up for clarity. First of all, the proxy_ssl_server_name and proxy_ssl_name is not needed in your setup, as NPM is not proxying to a http host. Also, you should disable "No TLS Verify" in the Cloudflare tunnel configuration - if everything is setup correctly it should not be needed and enabling it decreases the security of your setup.
…On Fri, Jul 7, 2023, at 9:45 AM, Andrew Moore wrote:
Sorry to comment on a closed issue, but if anyone could provide some advice on getting this working, it'd be greatly appreciated. I'm still getting the `Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF` 502 error from Cloudflare.
I've read through the thread and have implemented the fixes as follows:
• Added "Origin Server Name" to Cloudflare tunnel config.
• Added `proxy_ssl_server_name on` and `proxy_ssl_name $host` to custom Nginx config.
Am I missing something?
Screenshot 2023-07-07 at 17 42 15 <https://user-images.githubusercontent.com/20435317/251805092-37cc9d7e-04bf-4cc7-b6ea-73a3bd64397b.png>
Screenshot 2023-07-07 at 17 42 07 <https://user-images.githubusercontent.com/20435317/251805095-2e2d3f9d-e8ad-4ee6-841c-deb9a24a80b3.png>
Screenshot 2023-07-07 at 17 41 31 <https://user-images.githubusercontent.com/20435317/251805097-f5c1cab3-4856-4e66-a46f-04d2bae468f0.png> Screenshot 2023-07-07 at 17 38 08 <https://user-images.githubusercontent.com/20435317/251805102-5a5dea42-34e9-436b-9bbe-c8166b7d7397.png>
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R4HQPJ63RIEKFKI2WDXPA4MBANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Thanks. The certificate I'm using is just a wildcard origin cert generated through the Cloudflare SSL/TLS dashboard. Covering *.domain.com and domain.com. Which I'm assuming is the right type? I've double-checked and both domains are included in the SAN. I've disabled the ACL in NPM and the issue persists. I'm keeping "No TLS Verify” enabled for now, just to make it easier. It should work without though, I agree. Edit: I've just tried using a Let's Encrypt certificate on the origin and enabling "No TLS Verify" in Cloudflare. Same problem happens. It's strange that I'm having so much difficulty with this. If I point the Cloudflare tunnel at an origin server with a self-signed certificate I don't have an issue. |
Check the NPM logs. What do you see in the logs when you attempt to connect to your web site?
…On Fri, Jul 7, 2023, at 11:57 AM, Andrew Moore wrote:
>
> It appears that you are using a Cloudflare generated SSL certificate. Are you sure that certificate is for the hostname "overseerr*"? Also, try to temporary disable the access list in NPM to confirm that it is not incorrectly blocking incoming requests from Cloudflare. Additionally, while those are not the cause of your problem, you should probably still clean it up for clarity. First of all, the proxy_ssl_server_name and proxy_ssl_name is not needed in your setup, as NPM is not proxying to a http host. Also, you should disable "No TLS Verify" in the Cloudflare tunnel configuration - if everything is setup correctly it should not be needed and enabling it decreases the security of your setup.
> … <https://app.fastmail.com/mail/Inbox/compose?u=b1c14013#>
> On Fri, Jul 7, 2023, at 9:45 AM, Andrew Moore wrote: Sorry to comment on a closed issue, but if anyone could provide some advice on getting this working, it'd be greatly appreciated. I'm still getting the `Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF` 502 error from Cloudflare. I've read through the thread and have implemented the fixes as follows: • Added "Origin Server Name" to Cloudflare tunnel config. • Added `proxy_ssl_server_name on` and `proxy_ssl_name $host` to custom Nginx config. Am I missing something? Screenshot 2023-07-07 at 17 42 15 https://user-images.githubusercontent.com/20435317/251805092-37cc9d7e-04bf-4cc7-b6ea-73a3bd64397b.png Screenshot 2023-07-07 at 17 42 07 https://user-images.githubusercontent.com/20435317/251805095-2e2d3f9d-e8ad-4ee6-841c-deb9a24a80b3.png Screenshot 2023-07-07 at 17 41 31 https://user-images.githubusercontent.com/20435317/251805097-f5c1cab3-4856-4e66-a46f-04d2bae468f0.png Screenshot 2023-07-07 at 17 38 08 https://user-images.githubusercontent.com/20435317/251805102-5a5dea42-34e9-436b-9bbe-c8166b7d7397.png — Reply to this email directly, view it on GitHub <#2696 (comment) <#2696 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R4HQPJ63RIEKFKI2WDXPA4MBANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: **@**.***>
>
Thanks. The certificate I'm using is just a wildcard origin cert generated through the Cloudflare SSL/TLS dashboard. Covering *.domain.com and domain.com. Which I'm assuming is the right type? I've double-checked and both domains are included in the SAN.
I've disabled the ACL in NPM and the issue persists.
I'm keeping "No TLS Verify” enabled for now, just to make it easier. It should work without though, I agree.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R2T6FKVHIKGTVWYOALXPBLZTANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Nothing at all! Which is strange. When connecting to the external domain (via Cloudflare) I see the following cloudflared errors:
However, proxy-host-15_access.log and proxy-host-15-error.log show nothing at all when accessing externally. When connecting locally, I do see access logs. |
The name overseerr.andrewmoore.io doesn't seem to resolve for me. Are you sure DNS is setup correctly? Cloudflare obviously needs to be able to translate that name into an IP before it can connect to your origin server.
On Fri, Jul 7, 2023, at 1:11 PM, Andrew Moore wrote:
>
>
> Check the NPM logs. What do you see in the logs when you attempt to connect to your web site?
> … <https://app.fastmail.com/mail/Drafts/compose/M1c1ae105b0b29cfa13724fec?u=b1c14013#>
> On Fri, Jul 7, 2023, at 11:57 AM, Andrew Moore wrote: > > It appears that you are using a Cloudflare generated SSL certificate. Are you sure that certificate is for the hostname "overseerr*"? Also, try to temporary disable the access list in NPM to confirm that it is not incorrectly blocking incoming requests from Cloudflare. Additionally, while those are not the cause of your problem, you should probably still clean it up for clarity. First of all, the proxy_ssl_server_name and proxy_ssl_name is not needed in your setup, as NPM is not proxying to a http host. Also, you should disable "No TLS Verify" in the Cloudflare tunnel configuration - if everything is setup correctly it should not be needed and enabling it decreases the security of your setup. > … https://app.fastmail.com/mail/Inbox/compose?u=b1c14013# > On Fri, Jul 7, 2023, at 9:45 AM, Andrew Moore wrote: Sorry to comment on a closed issue, but if anyone could provide some advice on getting this working, it'd be greatly appreciated. I'm still getting the `Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF` 502 error from Cloudflare. I've read through the thread and have implemented the fixes as follows: • Added "Origin Server Name" to Cloudflare tunnel config. • Added `proxy_ssl_server_name on` and `proxy_ssl_name $host` to custom Nginx config. Am I missing something? Screenshot 2023-07-07 at 17 42 15 https://user-images.githubusercontent.com/20435317/251805092-37cc9d7e-04bf-4cc7-b6ea-73a3bd64397b.png Screenshot 2023-07-07 at 17 42 07 https://user-images.githubusercontent.com/20435317/251805095-2e2d3f9d-e8ad-4ee6-841c-deb9a24a80b3.png Screenshot 2023-07-07 at 17 41 31 https://user-images.githubusercontent.com/20435317/251805097-f5c1cab3-4856-4e66-a46f-04d2bae468f0.png Screenshot 2023-07-07 at 17 38 08 https://user-images.githubusercontent.com/20435317/251805102-5a5dea42-34e9-436b-9bbe-c8166b7d7397.png — Reply to this email directly, view it on GitHub <#2696 <#2696> (comment) <#2696 (comment) <#2696 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R4HQPJ63RIEKFKI2WDXPA4MBANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: *@*.**> > Thanks. The certificate I'm using is just a wildcard origin cert generated through the Cloudflare SSL/TLS dashboard. Covering *.domain.com and domain.com. Which I'm assuming is the right type? I've double-checked and both domains are included in the SAN. I've disabled the ACL in NPM and the issue persists. I'm keeping "No TLS Verify” enabled for now, just to make it easier. It should work without though, I agree. — Reply to this email directly, view it on GitHub <#2696 (comment) <#2696 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R2T6FKVHIKGTVWYOALXPBLZTANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: **@**.**>
>
>
Nothing at all! Which is strange. When connecting to the external domain (via Cloudflare) I see the following cloudflared errors:
`2023-07-07T20:08:33Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" cfRay=7e32b857fa83777a-LHR event=1 ingressRule=3 originService=https://overseerr.andrewmoore.io:443
2023-07-07T20:08:33Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" connIndex=1 dest=https://request.andrewmoore.io/favicon.ico event=0 ip=198.41.192.77 type=http
`
… However, proxy-host-15_access.log and proxy-host-15-error.log show nothing at all when accessing externally. When connecting locally, I do see access logs.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R5SM2YCG2BYCBTSZUDXPBUNXANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
That'll only resolve locally, as it's the backend address. It resolves fine within the cloudflared container. |
Never mind, yes, I got confused there for a moment.
I would expect nothing to show up in the proxy*log file, as obviously the connection is failing. That log file will only show something once NPM routes the request to the correct virtual host. But there's another log file (I think default*.log) that shows requests that didn't get routed anywhere. Do you see anything in there?
…On Fri, Jul 7, 2023, at 1:18 PM, Andrew Moore wrote:
> The name overseerr.andrewmoore.io doesn't seem to resolve for me. Are you sure DNS is setup correctly? Cloudflare obviously needs to be able to translate that name into an IP before it can connect to your origin server.
>
That'll only resolve locally, as it's the backend address. It resolves fine within the cloudflared container.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5RZBCVTHKQBZ4D6ZZY3XPBVKPANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Nothing in any of the default log files. |
Next I'd try to see if Cloudflare is even connecting to your host at all. Setup a tcpdump on port 443 on your origin server. Do you see packets connections when you try to connect via CF?
…On Fri, Jul 7, 2023, at 1:29 PM, Andrew Moore wrote:
Nothing in any of the default log files.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R4SPBWHCQDCXFMI6SLXPBWRXANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
What does the following command return when you run it on your origin server:
echo -n | openssl s_client -connect 172.16.1.33:443 -servername overseerr.andrewmoore.io | openssl x509 -noout -text | grep andrewmoore.io
This should connect to your NPM instance and print the SNI.
…On Fri, Jul 7, 2023, at 2:11 PM, Andrew Moore wrote:
I can see the cloudflared container (172.17.0.4) making a DNS request for overseerr.andrewmoore.io and then there is a key exchange with NPM (172.16.1.33), no traffic at all after that key exchange.
Screenshot 2023-07-07 at 22 09 06 <https://user-images.githubusercontent.com/20435317/251861989-92d08570-7b22-4401-a4c9-2ba5b1b95d62.png>
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R66OTUW7GAT4VUQVE3XPB3RJANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Please see below for the output:
|
I'm not sure if the Cloudflare certificate is the issue. If I use a Let's Encrypt certificate, which I know works because my browser trusts it, and enable “No TLS Verify” on Cloudflare, the issue still persists. |
I don't see anything obviously wrong with your setup, it should work as configured.
Two ideas:
1. Retype the name into the "Origin Server Name" field. I had a situation where I copy pasted the name into that field and somehow included a space at the end. Somehow CF didn't detect or filter out that extra space and things were broken until I figured it out.
2. Temporary disable SSL on NPM (i.e. make it a http site) and switch CF config to http, too. If that fixes it, it would confirm that there's indeed some kind of issue with the TLS connection.
…On Sat, Jul 8, 2023, at 2:59 AM, Andrew Moore wrote:
I'm not sure if the Cloudflare certificate is the issue. If I use a Let's Encrypt certificate, which I know works because my browser trusts it, and enable “No TLS Verify” on Cloudflare, the issue still persists.
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R7XSAIFM4DZ4HF4IYLXPEVR5ANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I think we're moving in the right direction to finding a resolution, but it's not making a lot of sense to me. Changed the NPM proxy host to HTTP only, with the Cloudflare tunnel configuration to match, hitting http://overseerr.andrewmoore.io:80 Going to https://request.andrewmoore.io now returns a 404 from openresty, which I guess is a good thing as we're now at least hitting the web server. default-host_access.log shows: cloudflared log: Hitting the same endpoint from anywhere locally works without issue.
|
Your virtual host is called overseerr while CF is configured to talk to request. You can override this in http settings of the CF configuration.
On Sat, Jul 8, 2023, at 8:20 AM, Andrew Moore wrote:
I think we're moving in the right direction to finding a resolution, but it's not making a lot of sense to me.
Changed the NPM proxy host to HTTP only, with the Cloudflare tunnel configuration to match, hitting http://overseerr.andrewmoore.io:80
Going to https://request.andrewmoore.io now returns a 404 from openresty, which I guess is a good thing as we're now at least hitting the web server. default-host_access.log shows:
`172.17.0.1 - - [08/Jul/2023:15:11:34 +0000] "GET / HTTP/1.1" 404 183 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"`
cloudflared log:
`2023-07-08T15:14:05Z DBG GET https://request.andrewmoore.io/ HTTP/1.1 cfRay=7e39465e2dae71ed-LHR connIndex=0 content-length=0 event=1 headers={"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Encoding":["gzip"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Cache-Control":["max-age=0"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["82.71.18.148"],"Cf-Ipcountry":["GB"],"Cf-Ray":["7e39465e2dae71ed-LHR"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Warp-Tag-Id":["079b8679-7ae3-4f7c-88fd-1dc6b7be020b"],"Cookie":["__cf_bm=wPxweKQxRSBleQcIGufRvojriwwyJf4iO2_9ONFT2vo-1688828570-0-AaO+PNy2JZS1aed6aORjtA1VNeVSmURXSFqb+A89ZqKixd5Aqf5Ll+eBoG2GdhwR5w=="],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Google Chrome\";v=\"114\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"],"X-Forwarded-For":["82.71.18.148"],"X-Forwarded-Proto":["https"]} host=request.andrewmoore.io ingressRule=3 path=/ 2023-07-08T15:14:05Z DBG 404 Not Found cfRay=7e39465e2dae71ed-LHR connIndex=0 content-length=-1 event=1`
Hitting the same endpoint from anywhere locally works without issue.
`# curl -IL http://overseerr.andrewmoore.io:80
HTTP/1.1 307 Temporary Redirect
Server: openresty
Date: Sat, 08 Jul 2023 15:16:39 GMT
Connection: keep-alive
X-Powered-By: Express
Location: /login
X-Served-By: overseerr.andrewmoore.io
HTTP/1.1 200 OK
Server: openresty
Date: Sat, 08 Jul 2023 15:16:39 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
X-Powered-By: Next.js
Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate
Vary: Accept-Encoding
X-Served-By: overseerr.andrewmoore.io
`
…
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5RZDS4TJSHMOL2HBOTTXPF3FNANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Gotcha, that's now working with the additional HTTP Host Header. |
That was it all along. HTTPS is now working too with the HTTP Host Header configured. Thanks so much for walking me through this. Sorry the issue wasn't more interesting haha! |
No problem, glad you got it working.
…On Sat, Jul 8, 2023, at 8:42 AM, Andrew Moore wrote:
That was it all along. HTTPS is now working too with the HTTP Host Header configured.
Thanks so much for walking me through this. Sorry the issue wasn't more interesting haha!
—
Reply to this email directly, view it on GitHub <#2696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUBW5R2V7NSEACUZ6XRQVQTXPF5WVANCNFSM6AAAAAAV55G3CY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I was getting this for v2.10.3. I fixed this by updating the cloudflared config. Here is how it looks now. Previously I was using a wildcard rule to catch all subdomains but this no longer works.
I did not change anything in the npm advanced setting for each proxy host. |
Thank you, this configuration worked for me on NPM 2.11.1 version |
Environment
Issue
Mitigation
The text was updated successfully, but these errors were encountered: