AI Megathread
-
@Aria said in AI Megathread:
I’m just sitting here watching this with a look of vague horror on my face
Oh dear heavens, all the sympathy. That sounds like my worst nightmare.
I worked in FDA-regulated software for awhile. I’m sure @Aria knows this well, but for non-software folks: In safety-critical, regulated industries, it is a well-known fact—learned through bitter experience, horrific recalls, and lost lives—that it is utterly impossible to test software thoroughly enough to ensure it’s safe once it’s already been built. There are just too many edge cases and permutations. Safety/quality has to be baked in through sound design practices.
AI has no concept of this. You can ask it to “write secure code” or whatever, but it fundamentally doesn’t know how to do that. I sincerely hope it does not take a rash of Therac 25 level disasters to teach the software industry that lesson again.
-
@Faraday said in AI Megathread:
@Aria said in AI Megathread:
I’m just sitting here watching this with a look of vague horror on my face
AI has no concept of this. You can ask it to “write secure code” or whatever, but it fundamentally doesn’t know how to do that. I sincerely hope it does not take a rash of Therac 25 level disasters to teach the software industry that lesson again.
Yeeep. I’m not in medical research anymore, but I used to work in the office that monitored clinical trials for massive university hospital system, including training clinical research coordinators on how to maintain documentation to standard.
You do not mess around with people’s lives, livelihoods, and life savings. If you break those, there’s really no coming back. I don’t understand why we have to keep learning this, but I guess some tech bro billionaire and all his investors that can’t actually follow along with what he’s saying need the money to upgrade their yachts.
-
I work in fintech. We are involved with some big cranky banks. The current AI push driven by our CEO is several GDPR breaches waiting to happen and my security guy is sitting there pulling his… actually he has no hair so I suppose he’s pulling his beard out! I’m right there with him, both on the frustration and lack of hair.
Devs, infra, solution design et al, we don’t get paid to write lines of code, we get paid to write the right lines of code. That’s why we have PRs and reviewing them is where all the productivity maybe-gained is being absolutely-lost.
I’m not even on the dev teams, I’m in infra, and even I’m copping it from product people trying to push code to my repos now. UGH.
-
@Hobbie
LOLOLOL. I also work at a Fintech, in compliance, and the direction on AI is incredibly schizophrenic. They’re like WE LOVE AI THE CEO WANTS MORE AI…but also don’t put ANY of your work into public AI, and probably not even the proprietary ChatGPT rip-off we have internally, because you’re probably gonna violate personally identifiable information agreements. I just don’t use it and am doing all right. There have been some nice internal robotics enhancements that’ve come out of this whole mess but none of them have given me any confidence this stuff will take my job anytime soon. -
@Hobbie said in AI Megathread:
I work in fintech. We are involved with some big cranky banks. The current AI push driven by our CEO is several GDPR breaches waiting to happen and my security guy is sitting there pulling his… actually he has no hair so I suppose he’s pulling his beard out! I’m right there with him, both on the frustration and lack of hair.
Devs, infra, solution design et al, we don’t get paid to write lines of code, we get paid to write the right lines of code. That’s why we have PRs and reviewing them is where all the productivity maybe-gained is being absolutely-lost.
I’m not even on the dev teams, I’m in infra, and even I’m copping it from product people trying to push code to my repos now. UGH.
But don’t worry, the solution will be to train an “AI” to accept and deny the right PRs so that way it’ll eventually get it right and then you can work on more important things and let “AI” fill in the rest.
… Twenty years later, when maybe something marketed as “AI” learns enough to write proper code instead of parrot it. (And after “a few” lawsuits and payouts related to GDPR and other data leaks, company implosions all over the world, etc. etc.)
But I’m not going to hold my breath on this.
-
@dvoraen And somehow someone still needs to know COBOL.
-
@Pavel Which GenAI will almost certainly never be able to because there isn’t enough COBOL stuff out there for it to steal for training data.
-
@Faraday All COBOL knowledge is held exclusively by two men, both called Steve. They’re not allowed to travel at the same time, to avoid the risk of all worldly knowledge of COBOL being lost in the same incident.
-
@Hobbie As another person who works in fintech, I have been incredibly lucky on this score - I’m the most senior tech other than the CTO, and both of us are incredibly sceptical of LLMs, so while there has been the occasional push to Do More With AI we’ve been able to stand firm and not bring it in to general use.
And we both got vindicated this quarter when two potential clients (the biggest we’d have) both went “we consider any use of AI to be high risk and we don’t want client information anywhere near it” and we went “excellent that’s our philosophy too.”
That said I’ve been banned from talking about LLMs on our all-hands calls as it invariably turns into a 20 minute rant about why they are bad for our purposes and the calls are only mean to be 30 minutes long.
-
This English professor asserting that em-dashes are the biggest tell-tale sign of her students using AI is what I’m talking about when I complain about people picking on the dashes.
-
bring back downvotes
-
@Rathenhope said in AI Megathread:
two potential clients (the biggest we’d have) both went “we consider any use of AI to be high risk and we don’t want client information anywhere near it”
Just casually send me the names of these clients so I can get our CEO to try and sell to them, I really want to see that metaphorical ice bucket tipped all over him when he hears that.
-
FYI, AI detectors are still kinda crappy.
Can AI Detectors Be Trusted? The Authors Guild Put Five of Them to the Test
“Our test confirms that while a couple of AI detection tools accurately identify human-authored text, some commonly used consumer-facing AI detection tools are wildly inaccurate, which presents a major risk for authors. Moreover, these tools change constantly—updated models, shifting benchmarks, evolving AI outputs—and their accuracy at any given moment cannot be assumed.”
Pangram Flagged My Own Writing as AI
Regarding false positive benchmarks: “that benchmark tests pure human text and pure AI text under controlled conditions. It doesn’t describe real-world use cases”
-
I don’t agree with your conclusions. From the same article:
The results varied widely, and in some cases, dramatically.
Pangram and Originality.ai were the most reliable performers. Pangram returned 0 percent across all ten articles. Originality.ai returned 0 percent on eight of ten, with 1 percent on the remaining two. Both tools correctly identified every piece as human written.
Grammarly performed nearly as well, returning 0 percent on eight articles and flagging two at 7 percent and 9 percent respectively—low enough that neither would likely trigger concern in practice.3 out of 5 did okay? They are tools to use alongside human reasoning. But demonstrably they aren’t garbage.
-
@Tez said in AI Megathread:
3 out of 5 did okay? They are tools to use alongside human reasoning. But demonstrably they aren’t garbage.
One of the tools you’re saying did “okay” in the first article is the same one the second article raises concerns about. Do they sometimes work? Sure. Are they reliable across a wide variety of use cases? No.
-
@Faraday IDK, I’d just continue to call them imperfect tools. That just puts it at 1 false flag in 11 samples instead of it’s 0 false flags out of 10 samples. Still not garbage.
I think there are real issues in these tools, but I think it is a mistake to write it all off as garbage. Use it as a tool alongside human judgment. Try to use the best tool you can.
-
@Tez said in AI Megathread:
Use it as a tool alongside human judgment.
Obviously this is the correct course, but basically every institutional use that I have personally seen has been lacking in the latter section.
Which is a problem in terms of work culture, not the tool, yes but when you’re relying on the tool to make decisions that directly impact a person’s likelihood of getting a degree, or getting/keeping a job? I don’t think we should be using, much less relying on, imperfect tools for in those instances.
-
@Pavel said in AI Megathread:
Obviously this is the correct course, but basically every institutional use that I have personally seen has been lacking in the latter section.
Which is a problem in terms of work culture, not the tool, yes but when you’re relying on the tool to make decisions that directly impact a person’s likelihood of getting a degree, or getting/keeping a job? I don’t think we should be using, much less relying on, imperfect tools for in those instances.
This is my problem, yes. However, I attribute it to the tool as much as the work culture.
These things portray themselves as far more reliable than they actually are in practice. The UX also lends toward this reliance, displaying results like “31% likely to be AI”. That, again, is giving the misleading impression of a highly precise and reliable tool. And unlike a plagiarism detector, which can point you to the original, these tools are much more opaque.
You don’t have to look far on substack to find tons of authors (pissed off about the recent addition of Pangram to the site) pointing out false positives in their own work. Even a comparatively low false-positive rate can create massive implications when used at large scales.
-
@Aria said in AI Megathread:
I’m pretty sure we’re all just living in the plot of Wall-E now and I hate it here.
Wall-E and Idiocracy. Yup.
