AI Part I: Listening to my robot overlord
The duckoid and the death of thought work
As I was copy editing this post, I realized I never got around to several promised themes, so I’ve relabeled this as AI Part I. Future posts will pick up the dangling threads.
On a lazy, slightly warm afternoon, while the rest of the country enters a shoulder season (it’s always such in San Diego), I pointed Gemini at this blog and asked it what was the next logical topic for me to write about. Perhaps not surprisingly it suggested I write about AI as “the most direct and unambiguous promise of a future topic in the entire collection of sources”.
By way of context I can point you to the first, and only thing I’ve previously written, dedicated to the topic. I wrote it in response to the mindless babble about AI I was hearing from co-workers, often traceable to the latest public press about ChatGPT1. I think it holds up quite well, and while I might refine the prose a bit, the core message is still valid and still represents my position. I argue that ChatGPT’s highly fluent and authoritatively voiced responses2 subtly shift users from actively evaluating sources to passively absorbing information. Its seamless linguistic delivery reframes responses as concise conclusions, inclining users to treat them as what I call “revealed knowledge”. This is seductive but epistemologically worrisome. Furthermore, there’s a deeper concern: if ChatGPT normalizes a tone reminiscent of mansplaining, might its deployment disproportionately reinforce harmful communication norms if organizations embrace its usage uncritically? This impact is particularly salient for DEI efforts.
Naturally, with the economic impact of AI, essentially the shifting of all tech resources from productive activities to supporting the build out of AI, discussions of AI are saturating the opinionosphere3. Much of it drek, but some of it quite smart, and even a smaller amount of truly deep analysis of the various dimensions of concern. One we should all be concerned about is that the current approach to AI is one of scaling. To make AI more successful, you need to throw more data at it, and to consume that data you need more processors, and to run those processors you need more power. We’re largely out of power, or will be soon. It appears that the availability of data with which to train LLMs has also reached an end. Chips appear to be infinitely available, provided politics don’t get in the way.
In this post I want to discuss a few different themes. Primarily, how cybersecurity is being impacted and what cyber practitioners should do in response. Next I’ll discuss how AI fits into the diminishment of thought work and the great stupidification.
Before getting started, it’s worth saying something about the linguistic weight of the term ‘artificial intelligence’. It used to drive me to drink to hear how that term is tossed around, yet there’s no evidence that even the most advanced LLMs display anything understood to be ‘intelligence’. Rather, they remain tremendous engines of statistical processing and pattern matching. Perhaps in the end, that’s all we are and skeptics such as myself will be shamed into silence. However I’ve tempered my criticism of this question by comparing the AI situation to that of image generation, particularly from video games.
If you want to create photorealistic images, the ne plus ultra of imaging is ray tracing: using relatively simple math to model beams of light as they interact with objects so as to present the eye with an image almost identical to what real light and real objects would create. Ray tracing is a straightforward process, most of us learned it by designing simple telescopes using nothing more than a spreadsheet. This is still how modern optical systems are designed, though for the last 30 years it has become the domain of increasingly sophisticated optical design programs.
But to ray trace an image is computationally intensive, especially if you’re trying to generate 60 frames per second (or more) as required by games. While modern home computers can just now barely handle this, it far exceeded the ability of earlier computers. So video game designers came up with any number of very clever techniques to produce images that appeared photorealistic, yet used no ray tracing whatsoever. While many of these just pushed up to the edge of the ‘uncanny valley’, they were more than good enough, and in some cases quite excellent.
So perhaps modern LLMs offer an analogous situation. No thought, no intelligence, but output that is often indistinguishable from that a human would produce. If it walks like a duck and talks like a duck, it may not be a duck, but a duckoid.
If, for a second, we put LLMs to the side, some of my cognitive dissonance around AI stems from the simple fact that it’s been used for ages in cybersecurity. Almost every tool that consumes large amounts of data4 has a machine learning component, and has for many years. At a recent security conference I attended it was impossible to count the vendors that had ‘AI’ either in their firm or product name. I imagined those that didn’t sitting at the kids table in a small room off the vendor display area.
Precisely what form of AI these vendors are using can be challenging to discern. Most, I suspect, are using classic machine learning tools, though many appear to be training models specifically on attack traces hoping to enhance detection of malicious activity in your environment. Environmental data, particularly in large diverse environments like universities, is incredibly noisy though and will always remain challenging5.
I think some of the most interesting opportunities are around automation. I can imagine having all of the initial responses to a possible compromise orchestrated by a trained model, with far more sophistication and intermediate analysis than simply following a series of commands. Network configurations could be changed to isolate a host, notifications could be sent (including a preliminary assessment), external sources could be alerted, and aggregated log data should be presented for human review. Of course this may sound like I’m recommending an AI package as an alternative to humans - I’m not. Beyond requiring a human to oversee the entire process, including intermediary steps, someone has to do the ongoing work of continuing to train and adjust the model. There’s probably a law somewhere that states, “every introduction of new timesaving technology increases the staffing requirements of a team using it by 20%.”
Two critical challenges for cybersecurity deal with how to secure AI installations and products, and how to protect ourselves from their use in enhancing attacks6. I’m astonished by how naive so many of my brethren are on securing AI models. Sure, bring all your traditional network and host based controls to bear. As a colleague told me, “Look at the data pipeline and workflow and see where AI is layered into this. Most of these AI pen tests find that it's basic stuff like access controls that are the issue.7” But additionally, take the extra step towards threat modeling and ask, what is it you’re actually protecting? The real value, and thus risk, in AI models often isn’t even the software code - it’s more likely to be either in the weights or in the algorithmic secrets used in the model that enable greater computation with existing compute resources8.
Of course, most organizations aren’t securing anything cutting edge or loaded with IP, they’re simply enabling access to services from Google, Microsoft, and Amazon9. Which means, like many cloud services, they’re failing to recognize both their own obligations for cybersecurity as well as how little their employees or students care what they’re offering. Faculty and students use the public or paid versions of whatever it is they prefer, as they’ve always done with online services. Don’t believe me? Block access to ChatGPT’s public offering at your network border. Let’s see how that goes over.
For all these services, locally developed or procured, the missing piece I see at most schools is any sort of threat modeling. What are the dimensions you’re trying to secure? Is it as simple as preventing cheating? Model poisoning by hacktivists or simply bored undergrads? If you’re concerned about privacy, what are you doing about protecting prompts10? Are researchers using your LLM? If so, do you have any concern about inadvertent or malicious leakage of data? What happens if your training data turns out to be corrupt and requires deleting? So many models are used to develop code - what happens when someone uploads some ransomware code and asks “could you give me 15 variations on this?”11 Probably worth a discussion about liability with your counsel.
Diving into each of these is a rabbit hole of impossible things to think about; these require expertise and technical controls beyond most institutions’ capacity. Unlike the White Queen, I wouldn’t feel boastful about tackling them before breakfast. Clearly you should create that high-level policy for using AI (probably a variation on your existing AUP), but just as clearly, that’s a half measure at best.
The good news is that there is a lot of activity around AI security, but it largely is coming out of commercial and governmental organizations. Higher education is simply not a player in this space, outside of a few researchers12. I think the point I’m trying to make here is that securing AI isn’t something you or your team can just pick up. It’s going to take some thoughtful planning, targeted training and study, and will be as multi-dimensional as every other element of information assurance. From risk assessment, policy, and engagement with your community, to technology, data mapping, and traditional controls. It’s a new world.
I do want to say a few words about the impact of AI on thought work. The business world's embrace of AI as a tool to replace “thought work” devalues human intelligence and creativity, enshrines mediocrity, and accelerates the “Great Stupidification.” As I suggested in an earlier piece, replacing genuine intelligence with a homogenized imitation may be the most efficient way to bleed distinctiveness from your workforce. If competitiveness is measured by originality and quality instead of Wall Street's moods, then an AI-driven staff isn't brilliance—it's mediocrity by design.
Further, we have the false promise of AI as a panacea for complex organizational or technical problems. Let’s expand on my earlier example of using AI to orchestrate incident response to illustrate this. What would this mean? In some environments, a compromised host is simply shut down - it’s a simple on/off switch. In practice though, even this simple of an action requires considerable nuance. Imagine turning off life-sustaining systems in a hospital. Or a medical record system. Such critical infrastructure can’t simply be treated like a random person’s laptop. There’d be an established process for handling such infrastructure, and frankly, few organizations understand and control their critical infrastructure sufficiently to allow this sort of automation.
Even if that challenge is solved, you’d want to pull data from a variety of sources to help you understand the incident. Host level data, clearly, but also DNS queries, network traffic flows (internal and external to your network), user activities, similar data from systems that may have interacted with the compromised host, machines and accounts that made the same DNS queries, application logs for systems the host’s user may have logged into, and so on. As experienced incident response analysts know, the scope and variety of data needed to understand an incident can be diverse and contextual. None of this is impossible, but as you might imagine just creating the training data for such a system would itself be a major initiative.
What’s critical here is not just that this is difficult - I still think it would be worth pursuing - but you want people who have a deep understanding of each of these steps. You want your incident response handlers and analysts to use their experience and judgement during the process. With enough money and time I’m sure this could be 100% automated, but in circumstances that are beyond the training data, or when your environment evolves (or involves political sensitivities) you don’t want to be at the mercy of AI hallucinations.
It might be interesting to survey attitudes about the use of AI in the workforce and look at the results demographically. Is it truly just the ‘leaders and executives’ that see AI as a way to cut labor costs? Are staff, hands on keyboard staff, also as excited by how AI could help them13? If we look at what’s being targeted for replacement by AI, does it tell us something about its relative value in the mind of management? I saw some mindless pontificating the other day about using AI to write annual performance reviews. Dear lord. Ignoring how this codifies cultural biases into the review process and boxes employees into an opaque and kafkaesque bureaucracy of judgement, does it really tell us how little respect management has for the performance review process? In theory, a performance review is there to help guide the employee, a human to human interaction of employee and management. Now it’s a target for optimization - to lighten the burden on managers? Talk about the death of thought work. We worry that students are using AI to cheat on exams yet in the professional world we’re going to use it to cheat at supporting our own staff? Have we no way to improve this process other than abandon it to a machine?
Of course, “artificial intelligence has had much the same effect as Darwin’s theory. Both aroused in some people anxieties about their own uniqueness, value and worth.14” While evolution is no more a ‘theory’ than the sun rising in the east, AI has engendered an awful lot of anxiety. Perhaps anything that causes people to be reflective about their own uniqueness, value, and worth is a good thing.
When people regurgitate the latest press, don’t they ever stop and think that everyone else in the room has read the exact same thing? No, no they don’t. Talk about stochastic parrots.
If memory serves, ChatGPT was the sole publicly available “AI” at the time.
Cybersecurity’s obsession with system and user activity logs and network traffic logs is why it should be considered a prime mover in the ‘big data’ space. Even my smallest office used several orders of magnitude more data daily than our institutional analytical stores and warehouses.
I recall one IPS vendor when seeing our network tap run through their box saying “why are you running a traffic generator? I thought you wanted to test it on live data.” It was live network traffic.
Beating swords into ploughshares? https://pentestgpt.ai/.
Private conversation.
I debated whether or not to include this reference, but I think the good just outweighs the hallucinogenic. Aschenbrenner firmly lands in the Skynet version of our future with AGI (Artificial General Intelligence) birthing any day now. So you can read those portions out loud at parties for amusement. However, notice how he talks about the key algorithmic secrets that are so critical in this arms race. There are a lot of smart things in this piece about cybersecurity that are worth studying and thinking about. https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf.
I know of one school that, “in order to better train the model” is/was storing every prompt to their homegrown LLM in an online excel file. Training, sure.
Hyrum Anderson (Cisco) & Ram Shankar (Microsoft) are the best of the best. Their book is amazing: https://www.amazon.com/Not-Bug-But-Sticker-Learning/dp/1119883989. Other people I pay attention to are Perri Adams, Joe Lucas.
I suspect many people are excited about using AI to improve their productivity and quality of work - yet the majority of what we read about is how corporate leaders want to reduce headcount. The disconnect is such an indictment of late-stage capitalism.
Herbert A. Simon, Models of My Life.