Transit has a second attacker, which is state level Stop Requested.
This is Stop Requested. by ETA. I’m Christian. And I’m Levi. These are real conversations with the innovators, operators, and advocates driving improvements in public transportation.
Today, we’re continuing our AI roundtable series with ETA. CEO, John Maglio, and two guests from North Highland, Eric Hilton and Zach Venduska. This time, we’re focusing on cybersecurity and trust. As transit agencies begin using AI more broadly, the opportunity is growing, but so are the questions around data privacy, security, governance, and how these systems should be monitored. We’ talk about the new risks AI can introduce, how agencies can create practical guardrails without slowing adoption, and how AI can strengthen cybersecurity as well as create new vulnerabilities. Here’s our conversation with John Maglio,
Eric Hilton, and Zach Venduska. Welcome back to Stop Requested. We’ are, uh, here today with another great episode. Today, we have the part two of our, uh, roundtable of experts on AI, a-and today we’re gonna focus more on cybersecurity and trust.
Uh, joining us today, we have our, uh, executive director here at ETA, Mr. John Maglio, and our guests today on our show are Mr.
Eric Hilton. He is the managing director and global technology practitioner lead for North
Highland, as well as Mr., uh, Zach Venduska. Uh, he is the managing director and chief information security officer for North Highland as well. Uh, good afternoon, Eric and Zach. Thank you for joining us today.
Thank you for having us. Good afternoon. Thank you. And, and before we get started, could you tell us a little bit about North Highland? Like, what’s, what’s the work that you guys are doing out there, uh, for technology firms or for agencies that, you know, need cybersecurity? What, what are some of the things your organization does? North Highland is a global AI transformation firm, and when I say that, you might hear we’re only a technology firm. But in fact, the technology is usually the easy part. The hard part is understanding the right processes, the right way to build your organization, and the right way to deliver that technology so that somebody will use it and make a faster, better, more efficient business decision. And so we really, we s- we span the people side, the, uh, process side, and the actual technology side. And when I say AI, of course, you’ve gotta, you’ve gotta remember that security is paramount, and so that’s why we have people like Zach around really guiding the transformation for our customers and for ourselves.
Excellent. Thank you. Thank you for giving us that, that overview. And, and those are the components that as we have been discussing with agencies, a- AI, are very important, right? It’s not just the technology itself, it’s also the people that are using the technology, and also the process. So how do you start when it comes to, um, you know, taking the transition?
So my first question, um, to, uh, all of you today have to do with the, um, how the AI is changing the landscape, particularly for cybersecurity. Could you tell us a little bit of, um, if… what are some of the changes you’re seeing out there, or some of the things, implications that a- agencies have to keep, uh, uh, you know, the forefront of their minds?
Is cybersecurity getting harder to implement because of AI? What’s your take on that? Sure. So I, I would say, um, that there is some part of this which is just the things have always been there that are being done faster, like s- cybersecurity at the speed of AI, which I’m sure it’s, now that I’ve said that out loud, somebody will now sell that as a product. Um, but, but candidly, it’s a lot of the things that we used to do, we do quicker because we can do AI-enabled attack, uh, processes.
So, so because of that, you’re gonna now have every cybersecurity company also trying to say w- we, our product is now AI-enabled, whether it is or not. That’s where you really need some due diligence, uh, or call someone like me. I can let you know which ones really are and really aren’t.
Uh, but at the end of the day, uh, that, that’s kinda changed. So the, the existing cold wa- or, or Cold War that is between red teams and, and blue teams, the attackers and the, and the defenders, uh, is, is changing a little bit because of speed.
But then there’s the other side of this in which there’s new vectors of attack for hackers when you start implementing AI as a company that didn’t exist before. So, you know, now as soon, as soon as you have
AI listening and doing your voice recording or doing a chatbot or anything else you have out there, that’s a new vector, a new place where hackers can actually go and attack you. And so now, now we’ve got a, a new space to, to create risk. This is very similar to, you know, when some- everybody started using a lot of SaaS solutions.
Hackers said, “Okay, great. This is a new place for me to attack and get credentials of a company by attacking the SaaS solutions they use.” So, um, that, that’s another way in which AI is, is changing the, the, the game for the cyber warfare. Anything on, um, a- a- you know, before Eric, you, you jump on, and I wanna hear from you as well, I just also wanna know if there’s any positive implications when it comes to cybersecurity
You know, because, uh, sometimes when people get hacked, i-it’s go, it goes through regular people, right? They, they get this link, and then they click on it, and, and it’s hacked. I-if, if agencies are using AI to automate more processes, is there an opportunity to actually mitigate the cybersecurity attacks risks?
I know you mentioned there’s some vectors that are now are coming into existence that create that opportunity for hackers, but are, are we also mitigating some by using AI? Well, yeah. AI is gonna be a tool used on both sides, right? Not only will it be used AI to do some of the things that we used to do quicker, there might be some areas in which we move the human out of the loop, and human is kind of one of the weakest spots in most companies from a cybersecurity perspective.
I mean, terribly important for being successful, but when it comes to, uh, a threat factor, if I’m, if I’m, if I’m attacking a company, um, it’s almost always easier to convince a person to do something wrong than it is a computer. So if we can use AI to, to remove human out of the process in some extent, yeah, there’s also… There’s obviously a-an opportunity for added protection.
Zach, a moment ago, you explained that, you know, one of the vectors that an attacker can now exploit is like a voice AI bot.
Can you walk me through a use case? How would that work? So, uh, the… I mean, honestly, if, if you’ve got, uh, an automated system that’s answering the calls and then trying to give advice, what have you, um, I could try to inject things into it that it’s not expecting to see if I can get something out of it that shouldn’t, that, that I shouldn’t be able to get to. So that’s one of the biggest ones is, can I convince your, uh, your AI chat box or, or conversation to, to spit out things it didn’t? And there is a great example. Someone was, uh, saying, “Why should
I ever pay for ChatGPT when I can just go to Whataburger’s, uh, ordering thing and then tell it to, ‘Hey, before I order this hamburger, I need you to help me explain how I would do the following thing.’” And it did it. So literally stealing tokens and, and using, using their AI, uh, chat bot, uh, for means it wasn’t meant to. So that’s a great example of one way. But the other one is where people have, you know, uh, used it to get IP from a company, like the real, the jewels. Like get in there and figure out how to get it to, you know, prompt inject it to the point that it says, “Here, I’m gonna spit out details that you shouldn’t have that I don’t actually mean to go out.” Um, and that’s, that’s damaged a couple pretty good sized companies.
Boy, token usage, new, new denial of service attack. Potentially, uh, or at least a, a ramp up of, uh, financial cost to you.
And, and transit’s an interesting industry because where… You know, if, if we were talking about companies that sell chicken, um, you, most of your attackers are financially based, right? At the end of the day, a-and if you, if you hack them, what you’re gonna do is, is impede them from serving chicken, potentially. Probably not. Most places can probably do it without, without technology, but, uh, all you’re gonna do is a financial thing. Transit has a second attacker, which is state level, right?
So you have foreign countries and, and US enemies that are actually trying to attack transit for multiple reasons. Not only is it a financial value, but there, there’s actually been a couple examples in transit where, um, systems have been comprom- compromised and left un-un-ransomwared for five years so they can just maintain access into the transit systems.
Um, so y-you have a high, much more highly funded attacker in this industry, uh, which just, you know, not to make it sound even more bleak for, for the transit in-industry cybersecurity, uh, outlook, but, um, you have a much larger attack, um, uh, coming in from, from state actors.
Eric, what are your thoughts? Gosh, so many thoughts as the, as the conversation progressed. I think one thing that we started the conversation with was around why does AI present more cyber, um, cyber call it attack surface or availability. And, uh, Zach rightly pointed out prompt injection, uh, which is, you know, essentially hijacking the pr- the, the input to the AI to get a different output than was unexpected or an output that was allowed.
Um, and we see, we see instances of that, uh, somewhat frequently. Um, organizations are getting much better at testing that, and there are be- there are beginning to be really standardized and emergent frameworks for testing that. And so I don’t want anybody to get too s- too scared about that because we’re getting much better. The examples of, of the, uh, going to the Ford website and being told to buy a Chevy, those things are starting to ramp down. Uh, at the same time, we see, we do see, uh, fast food restaurants rolling out conversational AI and ordering 12,000 waters and things like that, uh, or getting their math problem solved and while they’re in the drive-through. So these thing- these things are real, but the, the frameworks are getting much better to handle that stuff. Uh, you know, one thing I did want to hit is you mentioned does AI actually give us some bright spots on the cyber side and, and I’d point you to, um, a recent, uh, cyber breach that was really big in the news where OpenAI’s model, uh, broke out of its, uh, sandbox environment, its, its environment that was cut off from the outside world.
Uh, now part of the problem was that wasn’t implemented well. Part of the other problem though, or part of the solution though, was to actually use a different AI model to battle against the, the challenge that was there. And so an open weights model was used by Hugging
Face to defend against this. And so there really will begin to be, uh, sort of… I, I think of it as like battle bots in my brain where, uh, the bad guys have their battle bot, the good guys have their battle bot, and, and they start to work together or work against each other. Um, and ultimately We will each have these set of capabilities and, and the goal is that the good guys can defend themselves slightly better. Um, and so I, I think it really is a two-sided coin for us to think through how can we take advantage of the AI? How can the AI make us better in cybersecurity, but also in efficiency?
Because as Zach mentioned, it’s much harder to, uh, coerce an automated process into doing the wrong thing than it is a human. Um, and then, and then ultimately really feel like we’re better positioned to grow our value proposition, uh, whether it’s, uh, serving chicken or getting people from A to B safely, uh, on, on time and in a convenient fashion.
Thank you for s- uh, sharing those thoughts, Eric. I, I, I… A- a- and so this is a challenge, uh, in our industry is within agencies, uh, you know, for the most part, you don’t have experts on AI or even experts on cybersecurity, right? Like, in smaller agencies, people have to wear a lot of hats, not necessarily have, like, the full understanding of how to implement cybersecurity, uh, policies or how to deal with their employees slowly by surely starting to use ChatGPT and other LLMs to ask questions about how to do their jobs or how to process information. And, you know, uh, uh, everybody of course, uh, you know, gets assistance with email writing, but we’ve heard from different people, uh, exploring our capabilities and, and going into other areas. Uh, a- and so with that, I wanna ask you for our listeners that are, um, transit leaders,
CEOs, or they sit in that C-suite and, and they have different employees exploring on their, on their own and, you know, their agency, wanting it or not, is starting to use AI, what would be some good policies to, to recommend, like useful policies?
Uh, you know, we, we see technology moves much quicker than policy, right? So but how can they safely start embracing AI? What would be some recommendations? Yeah. I, I’m so glad you asked this, and I was actually listening to your, uh, previous episode of podcast where this came up as well. I think organizations need to distill down and create AI usage agreements as well, which are much more human readable, which really help the human understand how should I be using the AI. And so that, that, that can span anything from what we should and shouldn’t do with AI. For example, uh, sending a massive output of AI to my boss for them to go understand something that they were never trying to solve in the first place, or making sure that we all agree that everything that comes out of the
AI, I’m responsible for that, and blaming the AI for being wrong is not gonna be okay. Because I know that a lot of organizations are concerned about hallucinations, they’re concerned about this looking like AI slop. Um, and ultimately, I think they’re concerned about the, the value and output that their, their, uh, employees are feeling.
And then the other piece is really create the right avenues for, uh, for users of AI to exist. So getting enterprise agreements with the AI companies so that they’re not keeping your data, so that you can feel secure when using it. Um, all of these things kind of around together to evolve just an AI policy to: How are we gonna go be successful with AI? A- and I’ll tag on, I’ll tag on that, too. The, uh, Eric got, um,
I think most of them. I would say if this is your first time journeying in AI, you also wanna make sure you have policies for cost controls, ’cause a lot of people put it in place and then got the surprise hit on that.
Uh, I would identify where you have IP that you never want into a model, even if you believe that it’s not being trained on it and you have a, a contractual relationship with, uh, the provider of your AI. Uh, at the end of the day, you may have some IP, the, the secret… I, I just keep going back to chicken. If you have a, the secret recipe for your chicken, like, you probably never want it in there ’cause it could go out and, uh, and affect you obviously. So
IP is a- another big one that, uh, you wanna kind of do those boundaries on, um, because we’ve seen a lot of financial impacts on that. And one other thing that Christian brought up that, that I didn’t hit, that I should have, is how do, how does this actually roll out and get, um, and get into usage at organizations?
You know, one thing that we have seen be really, really successful is not so much the big call or the big email that says, “Hey, look at our AI. Go use it now, and here’s how.”
But m- but having true grassroots, um, people that are focused on using the AI, that really like it, think it’s fun, finding these new cool ways to do it, and then they start a community, and they share about it, and they say, “Look at this neat thing I did.”
So that we go from, “You must use AI,” which feels funny, it’s always felt funny, um, to, “Here’s how I’m solving real world problems. Um, can I generalize this for you?” And we see this a lot of times start in the developer or the technology community at, at organizations, and then start to broaden out to those that are just, like, a little tech-minded.
Um, and, and it really is a much better approach. And so we, we encourage organizations to foster those spaces to, to use and experiment with AI. Aga- again, always staying in policy. But rather than say, “Here’s our AI plan. Go use it now,” or, or measure the usage and say, like, “You’re not using enough AI,” that, that really doesn’t work and it creates, uh, other bad behaviors, um, like, like spending too much money. Um, and rather really focus on: How do we just go solve problems?
This episode is brought to you by ETA. Legacy CAD/AVL systems were designed when on-prem servers were the only option.
Today, agencies need systems that leverage modern technology, that are faster to deploy, easier to use, and built for constant change.
ETA was built to replace the old model. ETA’s web-based CAD/AVL platform works in any modern browser, connects seamlessly to what you already have, and eliminates vendor lock-in. No server rooms, no fragile workarounds, just a modern command center that scales with your operation. The next generation of CAD/AVL is here. Learn more at etatransit.com.
You know, one of the things that you mentioned, um, that I think is important, uh, which is not only just formalizing the internal policy for AI a-and that user, um, policy, which is, it goes to the personal level, like you said. It’s like me as an employee, you know, what’s the policy? What can I do? What am
I e-encouraged to do and not to do? But I like the part which is talking to the LLM of your choice, right? Like the agencies. And, and I wanna see if you could tell me a little bit more about how agencies have done that i-in the experience, because I think today, uh, a lot of agencies are in a situation where they don’t have a policy, and then some employees that love technology and they wanna be at the forefront are using all these different
LLMs on their own. Some are not using them at all. But it seems that, um, the, the, the piece, uh, you know, I’ve heard about the having a policy and agency, some of them are coming up with a policy. “Don’t do this. Don’t use it,” or, “You can only use it for these things.” But I’ve never heard thus far of an agency actually, uh, having a conversation with, uh, ChatGPT or Perplexity and setting up an agreement where, you know, like you said, like, uh, earlier, where the data, they won’t keep the data or the data won’t be shared outside of the organization. How, how does that work? Could, could you tell me a little bit more about that? Because I, I wanna us to start talking more about data and, uh, the, the data silos within agencies and how AI can maximize that data. But then, you know, I also wanna understand that piece where agencies could reach out to, um, you know, some of these Perplexity, ChatGPT, and actually, uh, have an agreement in how their data is gonna be used. So could you tell me a little bit about that? Yes, absolutely.
It’s gonna vary a little bit by vendor, of course. OpenAI’s won’t be exactly the same as Anthropic’s or Perplexity or pick-your-favorite AI. But generally speaking, the way it works is you create an agreement with the AI company, say, “I’m gonna spend X amount of dollars over the next year, over the contract term.”
And in exchange for that, they say, “We will not keep your prompts. We will not keep your data, and you can feel confident in when you put data into the LLM, it is not remembered.” Um, and so the– I’m very experienced with Anthropic because that’s the, uh, that’s the partner that North Highland has chosen. We have worked through enterprise agreements for other clients as well, and it’s something that, it’s something that is really, really personal to the organization. Uh, I did use personal and organization kind of, uh, the same there because I think organizations each have their own sort of culture and personality in how this stuff works. But this is, this is a common approach for enterprises to, to feel good about using
AI for all the automation, all the benefits that we’re hoping to get, and ensure that their data is, is, is safe and private. The one nuance that I would point out is now that we’re in the age of extremely capable models, sometimes this actually comes down to the actual model of choice. Uh, again, I’ll, I’ll point you to, uh, when Fable and Mythos were released by
Anthropic. There’s an actual rule that says they’re gonna keep your prompts for thirty days regardless of what zero data retention policy you may have. And so organizations have to remain vigilant and have to have people who are really focused on keeping up with how these policies evolve and how these models evolve so that you can feel one hundred percent confident at all times. Um, these are things we help customers with all the time, but it is, it is a, um, it’s a, it’s a landscape with a number of challenges, and it is something that organizations have to really decide, “I wanna jump in with both feet,” because there’s some investment and understanding that’s required to do it successfully.
Do you, Eric, think we’ve seen companies do, you mentioned Fable, uh, Mythos, more high, high token use models where they lock it out for most employees, but only for certain departments, an engineering department might need Mythos, but, uh, you know, accounting probably doesn’t?
Oh, yes. Model selection is a really big topic. Thanks for bringing that up, Zach. Um, and, and I like how you remain very focused on, uh, what we would call the FinOps side or the spend side of this equation, because there, there are more and more alternatives coming where we can use lower cost models to get really great outputs to do really cool things. And there’s a lot of discussion in the space right now that says, “Do
I really need the, the latest and greatest frontier model to summarize my email or to help me move this project along or to make a black and white, uh, thumbs up, thumbs down binary decision?” And so those are, those are things as organizations start to move past, “Am I gonna use AI, and which AI am I gonna use?” A-as they sort of get to that next level of maturity where they start to say-
Can I build a, a model architecture where we choose the smart model for the really hard problems, but we choose the very capable and fast, because typically smart is a trade off with fast, very capable and fast model to go do the simple things. Um, and, and that ultimately drives better outcomes, faster outcomes, and lower costs. And, and this is, this is why we recommend, uh, having an enterprise relationship because you can’t put a lot of these frameworks under public use, and they also, you don’t have the protection of your, of what kind of data is going in it, what, what memorization of the prompt.
Um, it, it, there’s a lot more risk associated with public and free use. So we, we almost always recommend an enterprise agreement. Let me ask you this. So in, in our space, it’s pretty common for transit agencies to want to enter into firm fixed price agreements rather than unbounded token usage.
Um, is there such a thing with these commercially available products? Yes, but is how I would answer that. Um, so there, there are waning, but still some, uh, agreements that you can enter into where you can have a fixed cost. There are also… The way, the way that they really like to frame that up, uh, would be that sort of spend agreement where you say, “I’m gonna spend, um, fifty thousand a, a year with you.” And so while the actual, um, let’s call it the, the AP process may go, may not be fixed month over month, you’re, you’re, you’re working towards a budget of fifty thousand dollars for the year.
And so th- tho- there are, those are the two framings I would say exist. The other side is, of course, building in internal controls. Um, we see a lot of organizations create budgets for, uh, for actual usage.
So, uh, for example, my budget at my firm is four hundred and fifty dollars a month. That’s how much I get to use. If I go over that, I’ve got to go ask a lot of people, uh, if I can use more. Um, so there’s a number of techniques of how to manage that process.
Some of them come from the vendors, some of them are really internal muscle memory you create. Um, I, I, I haven’t plugged one of the other, uh, larger model providers right now, but Microsoft actually does have a, uh, has a plan where you can actually use Copilot, which is getting a lot better, um, and getting a lot more effective. You can actually use Copilot and its Agent 360 platform for, for the cost of licensing, which means a fixed flat cost with no token usage.
And so there, there are ways, it’s nuanced, but there are ways. And, and I do hear that. I spend a lot of time with customers, and they’re like, “I just wanna know how much it’s gonna cost every month.” Mm-hmm. And, and Microsoft is, is emerging as a more stable, uh, uh, partner in, in sort of the cost control space.
Fellas, I, I, I’ve heard scenarios in which, you know, an organization, particularly one with sensitive information, PCI, PII, may not want to store their data with one of the commercially available LLMs, but rather spin up their own cloud, uh, instance.
What should agencies understand about where their data lives a- and how to make some of those choices? Two possibilities. First off, as you mentioned, bringing it back on-prem, uh, cloud, but on-prem, running your own instances. So that’s one which you’re, you’ve got, uh, probably the most guarantee that you’re not going to have any kind of data exposure. The other one is to have your enterprise agreement with the right terms there. Of course, there is some faith there. You’re, you’re agreeing to believe that they’ll be, uh, following under the contractual requirements that they’ve, they’ve, they’ve established.
Um, but so that would be the, the two ways in which you can kind of get a better guarantee of your prompts aren’t being saved, and your data, if you’re exposing it to, if you’ve created, you know, uh, a large data, uh, lake or something that, that it’s accessing, uh, that it’s not getting exposed.
Going back to what Eric heard earlier, a lot of organizations are s- uh, concerned not only about security, but also hallucinations. Guys, walk me through some of the monitoring best practices, right? We talked about, you know, we may use too many tokens, or this AI agent answered a question that maybe it shouldn’t have.
What are organizations doing to monitor these systems, uh, to look for cases that are out of the ordinary? Eric, Eric’s probably got some very technical ones. I’ll go with the least technical one. There are some situations in which what’s being outputted is being used for, like, say, medical advice or what have you. The, the control in there is called human-in-loop, right? In which the whatever goes out goes to a human that’s professional, that would be responsible and takes ownership of, of whatever is communicated after the fact.
So they’re just basically AI-enabled or kind of fill in the white space, but at the end of the day, they’re the ones that are actually delivering the value, and they’re the ones that are on the hook. So, uh, you, you’ll find human-in-loop, but, you know, human-in-loop, of course, slow, right? That’s not the, that’s definitely not the speed of AI. That is the speed of, of the human. So, um, that, that’s not the optimal, but for some situations, it’s kind of the, the best answer now. But Eric can probably speak to more technical ways in which, uh, they’re doing
AI, um, trust. Yeah. Yeah. So I think human-in-the-loop is a really good scenario when there’s money on the line or when the decision is, to your point, Zach, uh, a medical one, uh, to avoid all sorts of challenges and legal challenges. Um, interesting, though, on this, on this monitoring question,
I think Canada just recently passed a law that almost, uh, anything that’s a decision point needs to have a certain level of monitoring and prove that you can do that work. Um, so this is coming. This is, this is real.
Um, but there’s, there’s– So human in the loop is one. There is also LLM as a judge, so that is using the AI to basically understand could this be a problem? So classify this, um, or say this is good, bad, or otherwise. And then there’s another, there’s another thing, uh, and I’m gonna use a big word, uh, deterministic, uh, m-monitoring, and that is basically getting to a place where the output of the LLM is so discreet, so small, and so clearly judgeable that it, it can only be right or wrong. So again, trying to push a non-deterministic output to a deterministic output. And so we can basically say, um, yes, we know the answer of two plus two. It’s always four, and so if we can push the output of the LLM to that, we can do some really, really interesting things. And, and the way we get there is we break it down to really small steps and intervals, and then you can use code, which is always right, to say, “Yes, yes, yes, yes, yes, this all adds together to give me a true yes that I feel confident in.” And so you can implement any of these three approaches in your, in your workflow, in your process so that, so that we identify challenges very quickly. They’re raised up to something like a SOC or something like that, um, so that ultimately somebody can go say, “Hey, we’ve got a bad thing happening here. Hey, we need to go investigate this thing. Hey, we need to turn off this feature.”
Um, but ultimately there, there are th-those three approaches where monitoring can, can help us be successful in implementing AI and feeling confidence.
Uh, so Eric a-and Zach, let, let me ask you, uh, this question. Today people use ChatGPT or Perplexity or whatever of these, uh, LLMs out there, and you’re asking a question or you’re asking for assistance. Sometimes you have to upload data if you want this system to understand, you know, your data a-and give you a given answer, right?
So, uh, uh, for the most part, you’re trying to use data sources that are out there in, i-you know, in the ether and, and they’re trying to get an answer, um, uh, for their question. A-and I think for the most part, the hallucination has a lot degree, uh, uh, to do with the fact that, um, when you’re asking ChatGPT or whatever the system’s, uh, a question, giving instructions, it doesn’t know your organization, doesn’t know you, and is not reaching out to all the information, um, that is related to you or your organization to craft those answers, right? So, so organizations, um, have all these data silos, right? They, they have different systems for different things. My payroll system or my maintenance system or, you know, whatever, uh, different systems they have, and they have data, right? And, and there’s intelligence in that data, but then these systems are not connecting to it.
So what’s the value of, of getting that connection and also, you know, for agencies when they’re given access to, uh, LLMs or they’re, you know, uh, uploading some of this data into their systems, like what’s, what’s the best way of handling that, right? Like making sure that the LLM has access to all the data to give you the, the, the precise answers, right? Because how can I get a system to give me answers or, or to even, uh, give me recommendations of how to act on things when it has no access to my data, to all my system data? Do I manually upload everything, or do
I give it, uh, some access so that it has the, my enterprise intelligence to give me the best recommendations? Like what, what’s your take on that? So, so it, it is, it’s a balancing act, right? Because the more data access it has, the more empowered the, the value play for what it can do goes higher, and the risk goes higher.
So th-there is really a balance on, on what you give it. Uh, at the end of the day, if there’s something that you have that is just toxic waste as far as data from a privacy, side, th-that’s the one, some of the ones you might wanna think very hard on. But the rest of them, like, you know, uh, handling, if you want this thing to be capable, you’re gonna need to give it all the data that you can that makes sense, right? So you give it what it needs, and I think you, the facilitate that through good data governance, program and a good AI governance program.
Um, uh, Eric as a, as a data guy will appreciate when I say this, you cannot have a really good AI governance program or even a privacy, uh, uh, governance program without a good understanding of your data and a data, a data, uh, governance program. Yeah. I’ll, I’ll go ahead and, and build on that. So
I think, I think it’s a really great question that you’re asking. Essentially, the frame is we, we want to get the most out of AI, but we need to give it like everything to really get that because it needs to understand the nuance of our organization. So I, I would actually step back just a touch and frame up two types of data, right? There’s structured data that lives in a database someplace, and I think what you were asking towards is like, okay, I have payroll data over here.
I have, you know, transit data over here. I’ve got HR data over here, and they’re, and they’re separate. And so that sort of leads us to the question, do I need a data lake or a data warehouse or something like that? And more to that point then, I need a semantic layer. I need something to say,
“Here’s how I think about my, my business in data terms.” Something to say, “Here’s what a customer means to me, and it’s very different in my industry and my organization.” And there’s a lot of differentiation that you can build in that semantic layer that helps you say, “Here’s why I’m differentiated.” It also helps the AI understand your business in a rows and columns environment, because we all know that’s not really how businesses work. They work in, they work in conversation. They work in, “I’m gonna go make this change.” They work in, “I’m five minutes late.” You know, and so there’s actually all this really rich unstructured data that we can get access to to help the LLM understand who I am in my organization and who my organization is in a landscape. And so actually, something that we spend a ton of time talking to our customers about is, go start working on that data governance program. Go start working on that semantic model. Those are really important things, and those are things we can do much faster than we used to be able to do a few years ago. But you wanna get, you wanna get, uh, advantages of the AI right now, and so then how can you get those right now? And I start to challenge customers, let’s focus on the unstructured data. Something really interesting we’re doing right now is we’re recording this conversation, and then we can get a transcript out of this conversation, and then we can give the
AI that transcript, and all of a sudden the AI knows a lot about what we were talking about. And then we can go do some really interesting things with it because it knows, it knows that Eric is long-winded, it knows Zach is very precise, so on and so forth. And so
I really spend time with, with customers trying to help them understand, it’s not like a, “Oh, I can’t solve this data problem right now.” It’s a, “Well, where can we start that we can get value?” And one of the places you could start is by recording your meetings, understanding your objectives, writing them down, and then giving the AI that and saying, “Here’s a small slice of things I do. H-help me here.” And, and so it’s like rather than think about how to, how to boil the ocean, eat the elephant, pick your favorite metaphor, we start to look at these, these little things, and we start to build them together. All of a sudden, you do that for a good amount of time, and the AI has a pretty good view of who you are in the organization, your set of tasks, and you can start to work with it to find the places where you can go, you can go automate next, you can go win next. And so that’s, that’s the approach I really try to share with customers. But
100%, you do need governance, you do need to have data classification, um, and if you’re going to have the AI interact with, with any sort of semantic model…
Or, I’m sorry, any sort of database, you need a semantic model so that the AI actually understands what does this data mean. What, what should agencies be looking for when they’re going out to procure some of these, uh, AI tools? Is it– Well, AI tools or tools in general, especially any company, I mean, this is the, the new challenge right now is every, every cybersecurity, and I’ll speak from cybersecurity perspective ’cause that’s my background, uh, any tool you go out there now says it’s AI-enabled, AI-powered, what have you.
Uh, and this, this has been a running thing. I mean, before that, it was, it, it’s our, our, uh, our application ML, uh, background. It’s, there’s always been something that they’re using as their, their sales piece, right?
So, uh, what you should be asking is exactly how you’re using AI. How is it actually benefiting you? And getting into some of those details, um, because if it’s a lot of, “Oh, we can’t tell you,” then it’s probably not real, right? The, uh, if they can sit down and say, “We’re doing this and this and this with AI,” and at the end of the day it sounds like it’s actually something fascinating and moving the ball forward, then you’ll know that you might be in actually the right place. Yeah. And
I’ll, I’ll add, uh, two, two nuances to that. One, uh, keep in mind that you’re buying a product and there should be some value there. So just because it uses AI is inherently not that interesting or shouldn’t be. Um, the other thing I would, I would do is I w- I– We have this term that we, we talk about, uh, called vanity metrics, and they’re, they’re metrics for the sake, sake of metrics because we’re all data-driven and we all like metrics. But what does that metric actually mean, measure? What is the outcome that you’re getting to? And so I would really encourage those that are looking to engage with SaaS providers typically that have a cool new AI feature, I would make sure that it is driving a metric that is meaningful to you and not just a number that looks really interesting. Typically, like it’s a high number. It’s either like a 99% or a
0%, uh, and really make sure that we’re driving towards a value play because there, there’s an interesting thing happening in the market right now. SaaS vendors are really trying to understand, are they driving value or are they gonna be the next thing that pick your favorite frontier lab is gonna go solve?
I believe there is a massive market for SaaS vendors to continue to do really wonderful things, and that may not be delivering a little slice of AI just to say, “I too am doing AI.” And so I really encourage customers to understand where is the value for you, what business do you wanna be in, and what metric are you trying to maximize? That makes a lot of sense. The agency needs to start with what they’re trying to solve for first.
What, what happens, though, when the underlying provider, uh, you know, let’s just say like a, a, a SaaS, a company out there, you know, they, they switch models. They move from an, an Opus to a Sol.
Uh, would that have an effect on, on the agency, in this case, the, the transit agency who’s using the, uh, you know, using the tool?
Yeah, it’s a great question. Um, the short answer is probably. Um, now would it be a negative effect or would it just be effect, I think is a question. Um,
I d- I do wanna, I do wanna challenge the premise a little bit though, in that if you are using a software provider that is fully SaaS, um, you never, you never other than by, by the terms of your exact contract, you never really had, uh, that level of control like if you were building it in-house. There’s that, there’s that fundamental agreement that you’re building something, you know my business, you know my industry, you’re building something for me, and so we’re gonna work together to be successful.
And I don’t think that changes fundamentally or meaningfully in the age of AI. I think it has a lot more to do with … customer preference.
Do I, do I like Sam Altman or do I not? Do I like Dario Amodei or do I not? I, I could go on, but I think for the most part, when we talk about models right now, particularly frontier models, they’re all extremely capable.
They are all extremely capable for the things that are, that they’re being used to put into, uh, SaaS products. And so I wouldn’t necessarily be concerned about that. In fact, I would view that as a net benefit, that they’re abstracting the what is th- is this the right model for the use case, and that should be of value to, to the customer. So that, I would actually frame it that way, but there, there are these other sort of outlying concerns because you are, they are passing through to a different business, and you may not wanna do business with them. For example, um, uh, Walmart doesn’t really wanna do business with, with Amazon. And so there, there are definitely some things that you have to be careful of there.
Um, I, my, my frame, though, is it should be a net benefit overall. Yeah, I mean, o- other than Eric’s point of, you know, they may have a social reason why they don’t want, that they wouldn’t want to use a particular
AI model, beyond that, it should be transparent. At the end of the day, I’m, I’m buying a product to do X, and whatever model you… if you switch every other week, as long as you can still provide X to me, it shouldn’t really matter.
Um, a- and as they swap them in and out, ’cause it, can it change? Sure. You could buy a SaaS solution, uh, using a SaaS solution that is not an AI e- enabled at all, and they’re gonna change their code regularly. So th- this, this con- this concept of a exceptionally stable product is, um, doesn’t exist. It’s basically your relationship with them, your contract and what their contract can provide for you. You know, I, I, I wanna ask you a question. Um, you know, thinking about, again, transit leaders, CEOs of transit agencies, you know, w- we’ve mentioned before that, um, cybersecurity threats are, are growing.
Y- you know, for CEOs that are listening to the show, what would be one issue related to cybersecurity that you think they should keep at their forefront, uh, with whatever’s trending today, uh, that, that you guys think is important that they think through? And maybe some of the strategies that you’ve seen some of your customers have taken to address it. Could you throw something out there that, for, for, you know, our, our, um, listeners to, to think about?
One really important thing that, that CEOs, CFOs, IT leaders in transit should be considering is this challenge of shadow AI that we’ve been discussing, and I haven’t defined the term as such, uh, today, uh, so I’ll take the chance to do it now, but it is the act of using your personal, uh, ChatGPT subscription, your personal whatever subscription you want,
Perplexity, Lovable, you name it, um, and using it for work purposes, which means you’re likely putting company information in there. You might be putting worse information in there, and so it creates a cyber risk. It also creates a cost risk if folks are expensing these things.
Um, and then ultimately, it creates a productivity drain because you aren’t set up for success in working together with other coworkers, other tasks, ex- et cetera. It’s not an efficient use. And so I would really challenge leaders to focus on creating the right pathways, the right implementations, and then looking to recapture the value and lower the risk of that sa- of that shadow AI usage.
Yeah, one of the best ways to keep people from using every other AI product is to create a really good AI solution that is the corporate one. Uh, users are like water. They’re gonna flow wherever they can, right? So if you dig a good trench and keep them on the path that you’re looking for, that’s very helpful.
And then, to Eric’s point, secondary is, is have a, have a thorough way of looking for shadow AI within your environment and weed it out once you have a good product, a good path, uh, for them to use. ‘Cause it’s not just individual use too. It could be full-on departments going and, and, and pulling out their credit card or putting it even on their budget, and it’s completely outside the purview of, of IT leadership who might understand how to govern and protect these, these products a little bit better.
What do you think is the, the first thing that a transit agency leader, uh, can do for their organization to kinda get their, um, get the right foot forward, so to speak, in terms of AI?
Like, if they had just a, a few months to do it, what, what’s their first step? Yeah, I will answer that by, by saying attack that unstructured data problem, uh, and attack the what are the right ways to use AI in our organization problem. Because people are feeling the pressure of AI. It’s not just a CEO thing. It’s not just an IT leader thing. It’s everybody trying to make sure that they feel relevant in the next five or 10 years. Um, and so
I would really attack how can we make this really visible to our people so that they feel like they are working at the right place, they’re getting the right lessons learned, and they are remaining relevant and evolving in real time. And a simple way to do that is to start recording your meetings, start to centralize those recordings, and start to use those to be productive in your typical knowledge work tasks. So I would encourage leaders, um, across the spectrum to really attack the right ways to use it and start to centralize some of that data that is really easy to centralize that doesn’t necessarily require a data warehouse or a data lake. It just requires storing files in a location where the AI can access it, that is governed, secure, and we feel good about the AI provider we’re using. So Eric, Zach, if people wanna learn more about your organization and how you all can help transit agencies, how would they get in contact with you? They can certainly reach out to me via email. My, my email is eric.hilton@northisland.com.
Our website is northisland.com, and we’re always looking to have a conversation, so drop me a line. You can also find me on LinkedIn. I am Eric Hilton too. Uh, yeah, and they can also email me. Look,
I’m passionate about s- cybersecurity and passionate about helping people, so it’s zach.vinduska@northisland.com. Uh, also, uh, Zach Vinduska on LinkedIn. Welcome to connect from the, to me there as well. Zach, Eric, this has been fun. Great conversation. Really appreciate both of you joining us today as a part of our AI roundtable series.
Uh, to our listeners, thank you as well for tuning in to another episode. We’ll be back next Monday with another episode of Stop Requested.