Kevin Fenzi: infra weeksly recap: late October 2025

Scrye into the crystal ball

I didn't do a recap last week (because I was on PTO on friday and monday) and thought about not doing one today either (I was on PTO friday/yesterday), but I thought of a few good items to talk about. :)

Fedora Linux 43 released

Of course Fedora linux 43 was relased, you should install/upgrade to it today.

I typically upgrade all my machines at home (that aren't running rawhide) the week before release. I did that this time with all of them except one. On those other machines f43 was a typical nothing burger, no real problems everything working as expected.

On my main server however I held off on the upgrade for now. This is due to:

  • I run my own matrix server, using the matrix-synapse package in fedora. Sadly, this package has had issues in F43+ due to python stack changes. As I understand it, it uses pydandic, but via a v1 compatibility mode, which changed a bit due to python 3.14. Luckily the Fedora maintainer worked on a patch to move it to v2 and worked through all the tests. It's merged upstream now. I expect f43/rawhide builds soon.

  • There's some issues around postgresql. f42 uses postgresql16, but f43 provides postgresql18 by default. You need to upgrade through 17. I went ahead and just did this on f42. (both 16 and 17 are packaged for f42). So that should be ready for the upgrade to 18 now.

  • dovecot changed it's config file around a great deal. I still need to port my f42 dovecot config to the new version before I upgrade.

So, hopefully all that will be handed soon and I can upgrade that last server.

tcp timeout issues

The tcp timeout issues we are seeing between vlans in the new datacenter ( https://pagure.io/fedora-infrastructure/issue/12814 ) continues to vex.

Networking have tried a few things, I think it might be better, but we have not come up with a complete fix yet.

However, I did find another interesting datapoint. Moving our proxies to use port 8080 on backend kojipkgs servers (going directly to httpd there) instead of port 80 ( varnish ) has seen no failures.

So, it's looking like some kind of traffic issue with port 80 flows. Networking is trying to find anything that would just be affecting that.

Power news

Got 3 power9 servers setup and processing copr builds now. This should help with copr ppc64le build capacity, and it also allowed us to test/configure the 'fedora-isolated' vlan that is going to have all the things from the rdu2-cc datacenter moved to it in early december. (This includes pagure.io).

We now (finally, I hope) have a configuration for the power10's that will work for our needs. One of the two is setup this way now. I have created all the lpars and next week hopefully can get them installed. Once those are installed, I can move 1/2 of the existing buildvm-ppc64le's over to it and we can reconfigure the first server. This should allow spreading the load between the two and allow some more resources for them all.

Secure boot signing work

I finally have a ansible pr to setup the siguldry pesign bridge. Hopefully I can land that next week. This will move us from the current secure boot signing (a smart card on one builder) to using sigul (The thing that signs all out other stuff) doing the signing. We can then just configure builders we need to sign, no hardware changes needed. This will also now I sure hope allow us to setup signing for aarch64, something thats been in progress for like 6 years.

Stay Informed

Get the best articles every day for FREE. Cancel anytime.