This is interesting and scary at the same time.
Reading through the various descriptions - I pick up on a few details. It still comes down to a connection that can’t be established, which erroneously gets described as no Internet connection. There were signs of DNS not resolving, though all those URLs where https logically. Meaning there is both a DNS resolve and an SSL validation going on. To that extent it sounded like the ADSK team tried to reset certificates, which would be in line with SSL validation.
Chris said that the second Flame failed similarly the day after. When they were first built, did the second Flame maybe get built the following day, explaining the failure time offset?
One possible root cause - the licensing modules try to establish an https connection to their server and are not getting through. It could fail at the original DNS resolution, which can be circumvented with /etc/hosts. But that type of error, barring firewall or external changes, doesn’t fail without something happening locally. So less likely.
But once the DNS is resolved, the next step in setting up the connection is to check the SSL certificate of the host. If the certificate itself, or any of the intermediate CA certificates has expired, the connection can fail. Lets assume ADSK stays on top of their certificates, as not doing that would have widespread impact, as well as all the intermediate CAs.
I was doing a search on this topic, and there may be a local cache of some of these certificates. If that cache somehow got stale and not update correctly (and there have been hints that this can happen, though no specifics), that could explain why certain Flames suddenly out of the blue fall over - an intermediate CA cert that was installed during original build hit is expire date. And until that is resolved, any https connection that goes to hosts that use that certificate provider will become inaccessible. Mind you not all https connections, only those with this CA in common.
An OS rebuild from scratch would resolve that issue, as it (a) blows away that cache, and (b) seeds it with current certificates.
Next time this happens, you could try these commands:
sudo update-ca-certificates -f
and also these to check the certificates of those ADSK hosts
echo | openssl s_client -connect ‘<HOST:PORT>’ |grep -B 2 -A 5 “Certificate chain”
openssl s_client -connect '<HOST:PORT>' 2>/dev/null | openssl x509 -noout -dates
Those may not be the exact Rocky commands, so some research may be required
Just an idea worth trying.
