• Knock_Knock_Lemmy_In@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    arrow-down
    1
    ·
    1 year ago

    Run something with a 70% failure rate 10x and you get to a cumulative 98% pass rate. LLMs don’t get tired and they can be run in parallel.

    • jsomae@lemmy.ml
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 year ago

      The problem is they are not i.i.d., so this doesn’t really work. It works a bit, which is in my opinion why chain-of-thought is effective (it gives the LLM a chance to posit a couple answers first). However, we’re already looking at “agents,” so they’re probably already doing chain-of-thought.

      • Knock_Knock_Lemmy_In@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 year ago

        Very fair comment. In my experience even increasing the temperature you get stuck in local minimums

        I was just trying to illustrate how 70% failure rates can still be useful.

      • jsomae@lemmy.ml
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 year ago

        I’m impressed you can make strides with Rust with AI. I am in a similar boat, except I’ve found LLMs are terrible with Rust.

          • 𝕛𝕨𝕞-𝕕𝕖𝕧@lemmy.dbzer0.com
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            i find that rust’s architecture and design decisions give the LLM quite good guardrails and kind of keep it from doing anything too wonky. the issue arises in cases like these where the rust ecosystem is quite young and documentation/instruction can be poor, even for a human developer.

            i think rust actually is quite well suited to agentic development workflows, it just needs to mature more.

            • Log in | Sign up@lemmy.world
              link
              fedilink
              English
              arrow-up
              2
              ·
              1 year ago

              Ah, my bad, you’re right, for being consistently correct, I should have done 0.3^10=0.0000059049

              so the chances of it being right ten times in a row are less than one thousandth of a percent.

              No wonder I couldn’t get it to summarise my list of data right and it was always lying by the 7th row.

              • Knock_Knock_Lemmy_In@lemmy.world
                link
                fedilink
                English
                arrow-up
                1
                ·
                1 year ago

                That looks better. Even with a fair coin, 10 heads in a row is almost impossible.

                And if you are feeding the output back into a new instance of a model then the quality is highly likely to degrade.

                • Log in | Sign up@lemmy.world
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  1 year ago

                  Whereas if you ask a human to do the same thing ten times, the probability that they get all ten right is astronomically higher than 0.0000059049.