• Shayeta@feddit.org
    link
    fedilink
    English
    arrow-up
    31
    arrow-down
    6
    ·
    1 year ago

    It doesn’t matter if you need a human to review. AI has no way distinguishing between success and failure. Either way a human will have to review 100% of those tasks.

    • jsomae@lemmy.ml
      link
      fedilink
      English
      arrow-up
      13
      ·
      1 year ago

      Right, so this is really only useful in cases where either it’s vastly easier to verify an answer than posit one, or if a conventional program can verify the result of the AI’s output.

        • zbyte64@awful.systems
          link
          fedilink
          English
          arrow-up
          4
          arrow-down
          1
          ·
          1 year ago

          It’s usually vastly easier to verify an answer than posit one, if you have the patience to do so.

          I usually write 3x the code to test the code itself. Verification is often harder than implementation.

          • jsomae@lemmy.ml
            link
            fedilink
            English
            arrow-up
            4
            ·
            edit-2
            1 year ago

            It really depends on the context. Sometimes there are domains which require solving problems in NP, but where it turns out that most of these problems are actually not hard to solve by hand with a bit of tinkering. SAT solvers might completely fail, but humans can do it. Often it turns out that this means there’s a better algorithm that can exploit commanalities in the data. But a brute force approach might just be to give it to an LLM and then verify its answer. Verifying NP problems is easy.

            (This is speculation.)

            • zbyte64@awful.systems
              link
              fedilink
              English
              arrow-up
              3
              arrow-down
              1
              ·
              edit-2
              1 year ago

              Maybe it is because I started out in QA, but I have to strongly disagree. You should assume the code doesn’t work until proven otherwise, AI or not. Then when it doesn’t work I find it is easier to debug you own code than someone else’s and that includes AI.

                • zbyte64@awful.systems
                  link
                  fedilink
                  English
                  arrow-up
                  2
                  arrow-down
                  1
                  ·
                  edit-2
                  1 year ago

                  Literally the opposite experience when I helped material scientists with their R&D. Breaking in production would mean people who get paid 2x more than me are suddenly unable to do their job. But then again, our requirements made sense because we would literally look at a manual process to automate with the engineers. What you describe sounds like hell to me. There are greener pastures.

      • wise_pancake@lemmy.ca
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        Agents do that loop pretty well now, and Claude now uses your IDE’s LSP to help it code and catch errors in flow. I think Windsurf or Cursor also do that also.

        The tooling has improved a ton in the last 3 months.

    • Outbound7404@lemmy.ml
      link
      fedilink
      English
      arrow-up
      5
      arrow-down
      5
      ·
      1 year ago

      A human can review something close to correct a lot better than starting the task from zero.

        • loonsun@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          3
          ·
          1 year ago

          Depends on the context, there is a lot of work in the scientific methods community trying to use NLP to augment traditionally fully human processes such as thematic analysis and systematic literature reviews and you can have protocols for validation there without 100% human review