• KaKi87@jlai.luOP
    link
    fedilink
    English
    arrow-up
    2
    ·
    5 days ago

    I left a comment on your server but it doesn’t seem to be federating

    Oh, no ! I’ll have to look into that, thank you for noticing ! Any ideas how to debug that ?

    Terribly sorry about nerd sniping you!

    No problem haha, I’ve been wanting this for a while already ^^

    An easy way to fast track federation is using lemmy-federate.

    Interesting !

    Anyway, best of luck on your blog. If you’ve questions or want to discuss, feel free to send me a message!

    Thanks again !

    • kernelle@0d.gs
      link
      fedilink
      English
      arrow-up
      1
      ·
      4 days ago

      Lemmy’s docs show how to increase the verbosity of your logs, taking a look at them could point to possibly issues.

      Compare https://blog.kaki87.net/api/v3/federated_instances with https://jlai.lu/api/v3/federated_instances and I don’t think your server is federating properly. This graph of instance health shows federation stopped around 20/06.

      Since then, your new posts have federated to for example my server and lemmy.world, but 2 new posts from the last week did not make it to my server.

      I tried voting on this post with lemmy.world and the same post with 0d.gs which do not seem to federate but are visible on their respective servers.

      The strange thing is, in my Lemmy back-end, everything seems normal, and your server is telling mine it’s receiving correctly, but we can see it is not.

      • KaKi87@jlai.luOP
        link
        fedilink
        English
        arrow-up
        2
        ·
        3 days ago

        What am I looking for in the logs though ? All I see is “InboxTimeout”…

        • kernelle@0d.gs
          link
          fedilink
          English
          arrow-up
          1
          ·
          2 days ago

          /inbox is the endpoint for federation with ActivityPub, so definitely on the right track. Try increasing verbosity to get a full stack traceback to where in the code it’s bugging.

          InboxTimeout means your server is taking too long to process API calls to /inbox, I’ve had them before too due to misconfiguration of the reverse proxy. Make sure you’re passing on the right headers, especially the IP headers so they aren’t rate limited and the Accept header so your server knows to send json when requested.

          Troubleshooting federation gives many options for testing and checking your configuration, running through them might give your more insight.

          • KaKi87@jlai.luOP
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 day ago

            So, here’s the full error :

            lemmy-1  | 2025-07-14T00:34:32.098630Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=blog.kaki87.net http.target=/inbox otel.kind="server" request_id=a9c359f6-eb9d-4888-9ac2-55c1f5bb8e43}: actix_web::middleware::logger:
            Error in response: LemmyError { message: InboxTimeout, inner: InboxTimeout, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=blog.kaki87.net http.target=/inbox otel.kind=\"server\" request_id=a9c359f6-eb9d-4888-9ac2-55c1f5bb8e43", file: "src/root_span_builder.rs", line: 16 }] }
            lemmy-1  | 2025-07-14T00:34:32.098834Z  WARN Error encountered while processing
            the incoming HTTP request: lemmy_server::root_span_builder: InboxTimeout: InboxTimeout
            

            I’ve also been seeing this one :

            lemmy-1  | 2025-07-14T00:41:00.407762Z DEBUG HTTP request{http.method=GET http.scheme="http" http.host=lemmy:8536 http.target=/api/v3/site otel.kind="server" request_id=9aab8319-3f3b-4f4e-8530-3b3633afe2f1}: actix_web::extract: Error for Option<T> extractor: IncorrectLogin: IncorrectLogin
            lemmy-1  | 2025-07-14T00:41:00.427630Z DEBUG HTTP request{http.method=GET http.scheme="http" http.host=lemmy:8536 http.target=/api/v3/post/list otel.kind="server" request_id=ef746ac7-b9a3-4615-bb37-4c12f0ed0827}: actix_web::extract: Error for Option<T> extractor: IncorrectLogin: IncorrectLogin
            

            As for the reverse proxy, I’m using Caddy with the configuration provided here, and it’s supposed to pass those headers by default.

            The suggested curl requests work.

            I wish there was a tool that could just tell what’s wrong, like https://federationtester.matrix.org/ for Matrix…

            • kernelle@0d.gs
              link
              fedilink
              English
              arrow-up
              1
              ·
              2 hours ago

              Hmmm, some InboxTimeouts are to be expected. The second one could be the result of a scraper or bot. It’s still strange your federation list is empty. I’d take a look at the database, clearing the federation queue (public.federation_queue_state) is a good idea. ‘public.instance’ should be populated with servers (mine has 2836). Peaking into the log files of the database could give some more insight as well.

              A successfully federated incoming action will look like this: 2025-07-14T22:40:54.321151Z INFO HTTP request{http.method=POST http.scheme="http" http.host=0d.gs http.target=/inbox otel.kind="server" request_id=61bf0ea2-8df3-42ff-a433-3bec8b37047c}: actix_web::middleware::logger: 10.10.0.1 'POST /inbox HTTP/1.1' 200 0 '-' 'Lemmy/0.19.12; +https://lemmy.ml/' 0.158242

              Your outgoing federation seems to be fine now on my end, this is how my server sees yours:

              id 2794; ;domain "blog.kaki87.net"; published "2025-06-30T11:35:32.398830Z" ;updated "2025-07-14T00:42:32.758669Z" ;software "lemmy" ;version "0.19.11" ;federation_state: ;instance_id 2794 ;last_successful_id 46037 ;last_successful_published_time "2025-07-14T21:12:27.481507Z" ;fail_count 0

              You could try updating or reinstalling your back-end as well, lemmy is at 0.19.12.