Breadcrumb
Subscribe and Share

In Episode 85 of the Digital Velocity Podcast, Erik Martinez sits down with Pat Barry, President of demystifAI and a seasoned data scientist, to tackle one of the thorniest challenges in modern marketing: how to measure and forecast performance in a world full of fragmented data.

From McDonald’s digital tracking systems to direct-to-consumer (DTC) marketing attribution puzzles, Pat shares his cross-industry experiences using AI and synthetic data to fill in gaps, build more accurate forecasts, and drive smarter decision-making. They unpack the often-misunderstood limitations of Google Analytics, the data blind spots caused by Safari, cookies, and ad blockers, and the growing role of generative AI in turning incomplete data into actionable intelligence.

Whether you’re a Chief Marketing Officer building next year’s budget or a performance marketer struggling with conflicting reports, this episode offers deep insights into:
• Why most digital data is not as accurate as you think—and how to manage it
• How synthetic data can simulate realistic scenarios for forecasting and strategy
• Ways to test AI-generated forecasts for precision using statistical models
• What every DTC brand must know about attribution, consent loss, and multi-device journeys

Tune in to learn how to use AI as your data ‘time machine’—not just for insights, but for impact.

Contact Pat at:

Episode 85 – Pat Barry | Digital Velocity Podcast Transcript

Transcript

Episode 85 - Pat Barry

Narrator: [00:00:00] Welcome to the Digital Velocity Podcast, a podcast covering the intersection between strategy, digital marketing, and emerging trends impacting each of us. In each episode, we interview industry veterans to dive into the best hard hitting analysis of industry news and critical topics facing brand executives.

Now, your host, Erik Martinez.

Erik Martinez: Welcome to today's episode of the Digital Velocity Podcast. Today I'm here with Pat Barry, president of Demystify to talk about how we can leverage AI to help with measurement and forecasting.

Pat is a data scientist by training and brings a great deal of experience utilizing AI technologies to solve business problems with data. Pat, welcome to the show, man.

Pat Barry: Thanks Erik. Thanks for having me, man. I really appreciate it.

Erik Martinez: Dude, I have been looking forward to this. I say that to all my guests, just so you know. Not really. I've been looking forward to this conversation because, you've helped me with a couple projects, which we'll dive into a little bit in this [00:01:00] conversation. But before we get started, why don't you give the the audience a little bit about your background.

Pat Barry: Yeah. So, I've been working in, I don't know, tech or digital, if you will since about 2006. Actually I started my career in search engine optimization. I moved on, I worked at Discovery Channel for a little bit and built an SEO team for them there. And then I kind of pivoted to doing a lot more sales forecasting and reporting for their digital media sales teams at Discovery Digital.

So working on all their different websites and things like that. After that I moved on to Google, spent a little bit of time at Google as a principal analytical lead. Doing anything from analyzing well, not GA four, but Google Analytics data. It wasn't GA four back then. Doing forecasting a lot of analysis around the search auction. It's changed quite substantially. This was back in 2016 - 17. So, it's been a bit.

After Google, I've spent the bulk of my time after that in ad agencies. So worked at an agency called Critical Mass, which is a global digital only agency. I was able to work with a [00:02:00] lot of high-end clients like McDonald's.

From Critical Mass moved on to a agency called Mirum and I worked with Unilever. Partnered with the University of Illinois data science team, was able to prove out positive ROI and ROAS for social media influencer campaigns. It was rather complicated.

Erik Martinez: That's interesting.

Pat Barry: Oh man- getting in-store data from Unilever and their family of Kroger stores is something else. But you gotta get online spending data to understand kind of who's converting. That was interesting. And then worked with United Healthcare on their first ever personalization strategy and then executed it for one of their business units. So, complicated stuff. I use a lot of machine learning for those types of things.

Not generative AI, but I've been in the backend AI space, is what I call it, for at least seven or eight years. So it's nice to get some more of generative stuff.

Erik Martinez: So you've worked with a bunch of little companies, is what you're saying?

Pat Barry: Tiny guys.

Erik Martinez: Tiny guys. You did use the word high quality of McDonald's in the same sentence, [00:03:00] and I will say I do frequent McDonald's, but.

Pat Barry: It's.

Erik Martinez: Really,

Pat Barry: It's tasty, kidding.

Erik Martinez: You're a Chicago guy. You're supposed to be pro McDonald's, right?

Pat Barry: Of course. Well, they're based here. So, yeah their headquarters, they were a great client. Very interesting client.

Erik Martinez: Oh, I imagine. I imagine that had to be a very interesting business 'cause they're very scientific about what they do.

Pat Barry: Yeah. Really scientific. Maybe with how they make their food, but not so much on the marketing side, at least not five years ago. But interesting business. It's funny, I ended up leaving, Critical Mass, which agency I was working for when we were servicing McDonald's. But right towards the end I was gonna start to work on tracking for when you go into the stores now and you order on the giant screens. I was there when they were rolling those out and I was gonna be part of tagging those and tracking how people interact with those screens.

Erik Martinez: Oh, that's kind of cool. Well, you know what? I live in Kansas. And we're just getting those.

Pat Barry: Oh really?

Erik Martinez: I'm joking a little bit, it's funny when you say rollout of technology and you think [00:04:00] like, "Oh my gosh, they're rolling out stuff", and you're like, "No", they've been rolling it out for like six years, and Kansas is probably one of the last dates in the country to get it.

Which is totally true about how I feel about everything in Kansas. Digitally speaking.

Pat Barry: It's funny, I live in the Chicago ad area. I'm out in the Chicago suburbs. Those are like everywhere. They're in every single McDonald's. We used to have one. I saw the very first one. It was a prototype in our ad agency office. Big silver box. And you'd stand there and play around with it.

And we were gonna have to tag every single thing. Cause you can track all that. It's with web analytics, ironically. Everybody would think it's some sort of a fancy system, but no, at the time it was gonna be Adobe Analytics. You can easily set that up as if you're essentially tracking an app, like a web app is what it is.

It was super interesting, man. McDonald's is something else.

Erik Martinez: Yeah that's fascinating. Kind of fun. And then get to the world of direct to consumer and it's a little different.

Pat Barry: It is. .There's different challenges. I would say probably a little bit more difficult in terms of just [00:05:00] customer adoption. Everybody knows what McDonald's is on this planet, honestly. Whereas I wouldn't say a direct to consumer is as popular, but less well known in terms of, what that is and how people are using it for marketing.

Erik Martinez: But you just labeled a situation where McDonald's has actually gotten into the direct to consumer business. Right? You go order on your app and then you go pick it up.

Pat Barry: It's true.

Erik Martinez: That's a direct to consumer activity that is very interesting. Because when I came up in the direct to consumer business, it was mostly catalogs. And then it was ordering through the website and then we started building digital marketing programs.

Yes, I'm that old. And you know, retail and the restaurant portion of retail, it was still a very brick and mortar experience. You had to go there. I mean, the drive-thru was the thing, but you still had to go there and you still do. But it's amazing. Now you can order your food and have McDonald's delivered to your house.

Why you would pay DoorDash to do that is a whole [00:06:00] different discussion, which we will not get into today.

Pat Barry: It is probably for the better.

Erik Martinez: It's probably for the better, but let's talk about something that you helped me with recently,

Just so everybody knows, Pat and I embarked on this little side project. He was gracious enough to grant me some of his time to help me prepare for a talk on attribution. And during that conversation I said "There's really kind of three levels of attribution in my mind of ways to approach it. And all of them are fraught with some missing data".

If we're just using high level Google Analytics data, breaking it out by channel, doing something on a spreadsheet. If you compare your Google Ads data to your Google Analytics data there's a huge discrepancy and that gap is getting wider. So you have a hole there, and in that scenario, a lot of your channels, whether it's Facebook or TikTok or whatever it is, all those platforms are disconnected

Pat Barry: Correct.

Erik Martinez: From each other.

And that creates some problems. Right? [00:07:00] Then you and I talked about, "Okay, what's the level two attribution look like and what's a level three". Where Pat came in on this is to create this thing where we were trying to figure out how to fill in the gaps. So I'm gonna let you talk about how we go about filling in the gaps when data is missing. Because I think there's some misconceptions about people's data and how complete it is.

What has your experience been in terms of figuring out what your clients think is accurate versus what is accurate.

Pat Barry: Typically, honestly, the easiest thing that just about any client has is your basic website data. Usually with a free service like Google Analytics. But I've found, just having to explain to clients, they automatically assume this is one-to-one. This is actual like I had 10,000 people click this button I can prove that. And it's like, no, it's not. Google actually models the data out certain [00:08:00] point.

And again, the smaller the website you have the more accurate it is cause it can handle small amounts of data. But if you're getting to I think 10 million events within GA four. So you've got a large website and events now these days are page views. It's really anything. The page loads, that's an event. So you can get to 10 million events really quickly.

Because it's free, Google can't store all that data for you unless you wanna pay for it. You can upgrade your system and pay for Google Analytics four to get it to non-sample, but if you're using the free version, you hit 10 million events after that Google applies statistical modeling and they just model out the rest of the data.

So it's not actual data. And I think that's at least in, my experience, Kind of the first thing you tell the client is like, you're using all this digital marketing and most of it is either going to drive people to an app for whatever purpose or to a website.

Obviously yes you can drive people to other areas, social media, want, but I think I always kind of start there and explain the sampling end. 'cause a lot of people, " Oh, this is [00:09:00] exactly it". And again, for smaller websites, yeah, it's probably one-to-one.

But for larger ones, explaining sampling, and that's essentially when you hit a certain amount of traffic Google just it's forecasting and projections. They're just applying like a backend machine learning statistical model to say, "Okay, probably it's actually this many based on these events, what we've seen historically in the data, so on and so forth". So, it's kind of everywhere too.

A lot of people think like, oh my Facebook data socially, my meta, my TikTok, whatever it is. Those types of systems aren't usually aggregating at the user level. They're kind of taking it all and aggregating it together. I wish I could give you more information but a lot of those systems operate in a black box.

Meaning here's my money. Please go place these ads me and we'll give you reporting and you're just gonna have to take it for what it is. There is some of the documentation change it up here and then, but a lot of systems don't necessarily tell you exactly how they're counting.

Just a quick example from when I worked at Google. I can't remember if it was 2016 or 17, but Facebook [00:10:00] came out and said, " Listen, we've been over counting video views and how many people click through on a video ad, by three x". I mean, it was ridiculous.

So naturally being at Google, we seized on that. I had immediately put together decks of, "Hey, this is how we count. It's one-to-one. It was very transparent". But those types of things, you know, we had clients I remember at the time like, " All right, screw Facebook. We're gonna take our money away and give it to you guys 'cause we trust you more".

It's things like that can really erode trust. I think too, just the actual user journey itself. You know, somebody sees let's say a, Facebook ad watches a video, and they go do something else, and then they on their laptop, and then they pick up their phone, do another search. It happens to run into one of your paid search ads, and then they visit your website.

Stitching that data from the user on their laptop, visiting social to that same user using their phone and seeing a paid search ad that to any sort of system, unless you can stitch those together on the backend, that's like two different [00:11:00] people doing two different things. So it's difficult to just get the singular user journeys all together and understand all that.

And there's other things too, running ads within Apple and iOS and Safari, and I actually like kudos to them. They block like all of it. Safari just does not accept ads. It's great for privacy but for marketers it's just not, and that's something I've had teams deal forever like that's been going on for years now.

Erik Martinez: Yeah, and I think the stat I used in my presentation was something like, at least 12% of data coming from apple devices is completely lost.

Pat Barry: We've measured it out. I know in a couple of those spots I talked about in the ad agencies, like, we try to figure that out. 'cause we'd have to tell the clients like, "Listen, I wouldn't necessarily not run it on Safari. People use it". At the same time too I'm gonna give you like impressions and those might even be a little wonky because it just does not let us record that type of stuff.

I saw it with one client. We calculated it was like a 30% loss and it's hard. You know, they get mad 'cause you're the messenger, you're delivering the message, but it's like. " Listen, this is just what it is". And there's ways [00:12:00] around it, and that's the good news. it, this isn't, everything's lost. So while there's a lot of problems there are ways to kinda stitch all these different things

Erik Martinez: Well, this is my own experience, but in doing the research to kind of confirm that I didn't have confirmation bias. I also found that cookie consent is a huge source of data loss. Not as much here in the US depending on how you deploy it. Definitely overseas we've seen some massive decreases, but I've got one client who pretty consistently gets consent on about 60, 65% on their traffic.

Pat Barry: I'd actually say that's, pretty good. Typically half, that's a win anything above half is icing on the cake. You know, I thought for a long time too the cookie was gonna go away. I think I prepared for three, four years then I think it was the beginning of this year, like, Nope, it'll never leave. And I kind of figured it's tied to everything. It's in every disparate system. And I think we'll start to see some more privacy laws come into place, especially with AI coming into the mix, obviously. But [00:13:00] yeah, Europe has much stricter laws.

The odd thing is in all this experience, I've got never really had to deal with international clients. I've always been on kind of US based, I've been in healthcare really for the past like seven, eight years. There's much stricter laws with international clients like GDPR is real. You cannot - they will sue you for collecting data, and you're not telling people, you're just collecting it and you might not even know it, you can get in big legal trouble.

So it's much different overseas and really stitching the data together is the biggest.

Erik Martinez: And it's fascinating 'cause I know it's changing, but still really don't have the data collection systems to manage that sort of thing. We're really not there. It was funny 'cause I was traveling in the UK back late to 2009, 2010 for work. And you go to a restaurant there and they had the credit card chip readers right there at your table

Pat Barry: Oh man, ages ago.

Erik Martinez: Right. And it's only been really the last few years out here in the US that you see that as [00:14:00] being a predominant practice. Pretty much. You hand your credit card to somebody, they go disappear into some room, do their thing and come back. So it's really interesting.

Are there any other key data gaps that you see that are pretty common? 'Cause I came from a direct mail background and I can tell you that my colleagues who have also come from a direct mail background tend to have a bias towards direct mail data.

It's more accurate, it's cleaner, right? We know who we mailed, we can take a transaction and match it up. It's still an attribution window that we are using in that scenario. And it's funny though, I've had a few clients acquired by digital companies in the last few years, and they don't believe any of the offline data.

They think the digital data is hugely accurate. So I think part of our conversation today is just kind of dispel those myths and say, "Hey, you know what? You're not working with as accurate data as you expect". However, [00:15:00] statistically, if you have a lot of data, you can be reasonably precise.

Pat Barry: And I think I like your phrase reasonably precise because with direct mail there's other ways you can track it as well, especially if you want to see what they did digitally. You can add a QR code or a vanity URL but that's still not necessarily a hundred percent accurate. We'll never know full on, especially billboards used to get a lot of questions, "Hey, how can I track that billboard"? Like you put a vanity URL up there I can at least tell you when somebody came to the vanity URL and at what time. But if they're driving by a billboard, I have no idea when they saw the billboard. But at some point they came here.

I think a lot of it is just even the, small things, set up naming conventions for your UTM codes. If that's not uniform on your end and that's something you can control, then you're gonna have messy data. I think it's that's schema stuff. Cross device tracking is just not necessarily legal all the time, depending on what type of marketing you're doing. Healthcare, you can't do any of that at all. It's just [00:16:00] a big no-no. CPG type stuff,

For sure, I mean, it's a little less but just stitching it all together and really understanding like ad blockers. Okay, those are real, a lot of people are using them just outside of safari. Those are the types of things to look out for.

The good news is if you do have just data, you've got it. You're confident in it. There's a lot you can do is create synthetic data to kind of fill in those gaps. I don't know if we wanted to hit it quite quickly.

Erik Martinez: Yeah, go for it, man. Because I think that's really what we're talking about is how do I overcome some of these and there's gonna be brands that don't have the amount of data that some of the companies you've worked with do. And so they're working with really smaller data sets and then smaller subsets of that data depending on which channels that they spend most of their money and time on.

So synthetic data i'm gonna let the data scientists define this.

Pat Barry: Synthetic data, just in a simple world it's essentially fake artificial [00:17:00] data that statistically mimics your actual data. So for example, you're running an ad campaign and for some odd reason you're missing your impression data for like five, six 10 days of the last 90. What you can do is fill in those missing gaps with reasonable numbers based on the rest of the data set that you're seeing to kind of say, " Okay, what probably happened here". Now I've realized impressions is probably a terrible one 'cause if you're not getting your base impression data from your ad provider, then that's pretty bad.

At the same time, too, just as a high level example, it's really meant to, mimic the data that you already have there and to fill in the blanks just so you can run some sort of a statistical model that can handle the synthetic data that's, in there. It's used for forecasting and projecting, testing what you've done. Filling in gaps that while they'll be directional, it's a lot more accurate than zero than nothing, and not knowing what happened that day.

So it's a fairly [00:18:00] common practice. My teams would use it a decent amount or have to create synthetic data. You don't necessarily even have to use it for marketing. I know a lot of finance folks that like, " We're trying to get all this together and we're just missing cost items. Okay. Well, you have a rough idea of what those things cost. Yep". We can generate at least synthetic data, kind of fill those gaps and give you a better statistically sound idea, of what happened on that day, week, month, whatever it is.

Erik Martinez: We do that in our reporting. Our reporting system is set up in a way where we obviously take the data out of the platforms and Google Analytics four and all that stuff and we stitch it together. And you and I have talked about our ecosystem, but when we do see those outages, 'cause you know, nobody's ever had a pixel get accidentally blown away, or the recent one. I was talking to my reporting expert yesterday was a client updating their site, ripped out their GTM container and well, there's no event data for

Pat Barry: Yeah, your container is gone.

Erik Martinez: [00:19:00] Right? So the container's gone we're not collecting any data. And in those scenarios what we do is we go do those estimates. And we did create a version of synthetic data, but what you're talking about right now is really using AI to help create that synthetic data quickly and then we've got multiple use cases. So you gave one use case already which is, "Hey, we're just gonna fill in the gaps where we know we're just missing some information and we're just trying to estimate as closely as we can so we can do an analysis".

Then there's another use case, which is what you and I have been talking about, which is. In the budgeting and forecasting cycles within your companies. If you're trying to figure out, well, shoot, last year I spent, I'm just gonna make up a number, $10,000 on Facebook. I want to increase that to a hundred thousand dollars. I really don't have a good basis. for what that's going to look like in terms impact on a business.

And one of the things [00:20:00] you did for me was create a couple really cool prompts for creating synthetic data to kind of deal with that scenario. So let's talk a little bit about some of those other use cases that synthetic data can help us.

Pat Barry: The example you just gave, "Hey, we spent 10 grand, we got", let's just say, conversions. Just make it simple like we're looking to drive conversions, whatever it is, whether that's a click on an ad and go to a site or fill out a form. If we upped our spend by, in your, case, 10 x from 10 grand to a hundred thousand. How much would our conversions increase?

And so with AI now the prompts are fairly complicated , but if you need to be able to look at your data. Upload a data set into Chat GPT, like two years is usually the best amount to get a good forecast. but if you need something and you've got six months worth of data and you wanna project out the next two, three months, you can definitely do it. It just might not be as accurate 'cause you're not going back as far, but most people don't necessarily have two years worth of good [00:21:00] clean marketing data. And if they do, there's usually gaps. There's usually a missed context around declines in spend or increases in spend that your client made.

If we go all the way back five years to the pandemic era, spending was all over the place. I was working with a lot of hospital systems at the time and they had huge traffic spikes that we could have never predicted because nobody had to go get sign up for vaccines online before.

So there's, a lot you need to understand of past performance. But what else is gonna happen out in the world that can affect all your forecasting and outcomes? So a lot of it is you need to know your real data. Like what are all of my different variables and metrics - impressions, cost, clicks click through rate, so on and so forth.

Understanding what's called dependent and independent variables. So for example, your impressions metric, if you will, is a dependent variable. It's dependent on your spend. So your spend is, your independent variable spend is [00:22:00] whatever. It depends on the budget, depends on what you wanna do. Once you kind of understand what your independent dependent variables are, you can kind of start to project out your dependent variables.

Erik Martinez: Now you're getting all statistical nerdy on us. It's okay because it's really a germane topic. It's kind of dry. I remember sitting through statistics class in college and going, blah, and when I graduated from college, I had two professional level statistics courses that were like hardcore regression analysis type courses. And I remember at the time going, "Oh my god, my head hurts". But really what we're talking about is probabilities. Right?

What's the probability of something happening in this scenario? And so we're using real data to have AI help us create synthetic data that would mirror what the world would look like under various scenarios.

So spend being the independent variable might be, "Hey, I'm going from $10,000". Here's what we got. Here's what other industry statistics look like. [00:23:00] That could be an input into the prompt, right? And then say, " Okay, what does this look like if I expand it to a hundred thousand dollars?" And then I may ask some other questions like, what's the impact on my other channels? Is it gonna have a positive impact or a negative impact?

Pat Barry: And I know those prompts I wrote for you were crazy. I mean, these were literally like probably almost a full page of a word doc of what you had to tell it. And I think the one thing probably people don't think about is the model creating and then you wanna forecast it out, "Hey, if I spend a hundred grand more. What does that look like?"

Well, that the first thing you need to tell it is like you need to spend that a hundred grand over three months, six months. Don't spend any on the weekends because we know it's a lower time. With digital, obviously there's different devices, so you might wanna tell it like social media. I'm guessing this is still true. I don't have any updated stats from right now, but. Yeah. Most people are on their mobile phone when they're looking at social, frankly, so, you know, telling it all right, you need to now look at my [00:24:00] conversions between desktop and mobile, and you need to meter out the spend between those in some sort of rational fashion.

It's more understanding the granularity of your historical data and being able to explain it. I will say this, I know when we worked on our stuff, that like two months ago, which feels like ages ago at this point. Well, there's more models out now that can literally, do a bit more with less like information.

But having the AI, whatever your using Chat GPT, Claude, whatever your generative AI system is like, here's a sample of it. This is what all these metrics mean. Here's some extra context around the date range I gave you. You could upload whatever you want into chat GPT and tell it like, " I want you to create synthetic data on one in the date column. Give me another 90 days. And then just conversions." It'll just look at what you put in front of it and say it's about this. But if you don't give it more specifics, especially those dependent and independent metrics, because it doesn't understand those out of the gate. [00:25:00] So you have to tell it.

The more you get into the marketing data, as we both know, some of these systems give you, I mean, my God, like hundreds of different metrics for a single ad campaign, probably 80% you just, don't even need. It's just like you would never look at it. So even understanding the complexity of what those exact measures are.

Find what you need to measure and that you want to project out and focusing on those is key. And you should have a measurement strategy in place before you start any sort of campaign. And identify those key metrics that are the most valuable to you.

Probably something else I should have mentioned, a lot of people start out with like, "Oh my God, there's a lot here." Yes there is. You need to understand which ones are the most effective and which ones you need to track. So just getting through the volume of data can be a challenge for people as well.

Erik Martinez: And I think that's true. I've messed with lots of large data sets my entire career. Maybe not quite as large as some of the ones that you have worked with. But that's always the biggest part is like wrestling the data into an organized format where it's consistent and [00:26:00] where you have some potential holes that you have to deal with.

Either ignore them, 'cause they're small or you are like, "Oh, I've gotta fill that gap because we had a data outage for a week because the pixels got blown off the site because somebody deleted the container." Right? So once we have all that data organized, and let's say we've got really good descriptive data going back two years and we start loading in some synthetic data. What is the next step?

I think some people are probably sitting here going, "Okay, well that's all great and good, Erik, but you're creating a budget or a forecast based on not totally real data." How do we make sure that's the right way to go?

Pat Barry: So you need to test essentially I know we might have to nerd this up a little bit more, if you were just surely like, " I got three months worth of data. this is all I have - and I have to create a projection. There's a lot of different statistical methodologies you can just look at. The root mean squared error, the mean absolute percentage error. [00:27:00] I won't go into details. These are things you can run and test.

It's basically like we'll give you a score that spits out. It's a zero to one score. And again, there's more technical stuff to this, but the closer it is to zero, the more accurate it probably is. In some scenarios like that. I always encourage to try - especially with forecasting and projections. If you've got, hopefully, two years worth of data, why don't we say like - give me 2023 data. Let's forecast that out. Cause I've got the 2024 data and I can just see how accurate this methodology is based on assumptions I've told it, things like that.

So it's always best to test all this stuff first on real data. And I know with synthetic data, that's not the easiest thing to do. So test first if you can, on your real data. If you don't, when you're using synthetic data. There's different statistical methodologies that you can utilize.

A lot of these are actually baked into Excel. So you can do a lot of these functions in Excel. Most people don't like the outputs. They look like computer code garbage [00:28:00] unless you know what you're looking at.

Erik Martinez: Couldn't we also just have the AI do that for us too. If we knew which ones to employ" or just say, "Hey, employ the whole, you know, statistical analysis measurement."

Pat Barry: So yes you could, yes, in theory, upload your data to chat GPT and say, Hey. I've given you, 90 days worth of marketing data. so it knows what they're looking at. And then yes, can you run the root mean squared error, the mean squared error, the mean absolute error, and then the mean absolute percentage error. And yes, it would do it.

Erik Martinez: Yes, there is a test at the end of this episode.

Pat Barry: I was desperately trying to avoid saying those things. I can't, like, and if you Googled any of those, they go by M-A-E-M-A-P-E-M-S-E, and RMSE. They're simple definitions. This is kind of like statistics 101 type of stuff you learn right out of the gate.

And again, they're simple bellwethers to see, how accurate your prediction is. The [00:29:00] non-technical way to put it like where does it fall in this zero to one scale? And you'd kind of look at these four different scores and then decide like, okay, it's never gonna be perfect. This is exactly what it is. So you need to kind of know that going in, and especially if you have to explain these things to a customer, a client, or somebody internally like your CFO, like, "Hey, I'm fighting for more marketing budget. Here's what I came in with." Like, they're gonna ask, " Well, is this real? No, it's directional, but it's either I can show you this or nothing."

There's still some guesswork in it. And I typically tell people, and clients especially, they're always, "Well, AI - we're all advanced and all this." Like until humans are walking around with like chips in their head that will send their emotions and their decision making and how they got to this point to another database, which my team will then analyze. You're honest to God, really not gonna know. So you're always gonna end up looking at some sort of directional, like, all right, the compass is pointing northwest. Let's go walk that way and we'll find out really quick if we fall into a tar [00:30:00] pit or whatever. But I think that's what you've gotta wrap your head around is like, okay, make sure you write out your assumptions.

What did I tell the AI that this data is about? Did it use the right methodology? There's all kinds of different methodologies. And then testing, like how did the numbers come back? These look good. Okay, great. I need to be able to translate that into English, to the person that I'm telling this to and why I think this forecast and this synthetic data is correct.

So there's a lot more explaining the imperfections, but why you have confidence in the data, and why we think this is the best way to go about make X, Y, Z decision with our budget You know, where we're gonna move market spending.

Erik Martinez: I think you've highlighted that forecasting is a fairly complex discipline that has a lot of variables in it that can move either way and your best forecast are gonna have applied statistical techniques to historical data to make a prediction about the future.

I think where a lot of people [00:31:00] kind of get stuck is, " Oh my God, it's September and my first draft budget is due next week and I haven't started doing anything yet." Like I've been thinking about it 'cause I know it's coming up, but I haven't put the proverbial pen to paper to develop the budget.

And so Pat, oh my gosh. What you just talked about sounds like a whole lot of work and effort, even with AI involved in the process. So the question is, " How do I simplify that?" How do I start to maybe move down this path to make it a little bit better.

What are your recommendations? Like, " Hey, do this, do that, do this other thing." And if you start with those three things, that's really good.

Pat Barry: I would say in terms of forecasting is number one, like understand your data and the relationships with that data. We talked about this a little bit at the start, you know, spend to impressions is the obvious one. So however much you spend, there should be a direct correlation in the number of [00:32:00] impressions you get from that amount of money. So that's kind of one thing. Make sure you understand your data.

Doing a little bit of homework on just statistical methodologies, and I know this is gonna sound crazy using a simple average, what's the average? Sometimes I've done that because it's just the most damn accurate.

There's a couple other methodologies that I know we talked about that the AIs are very familiar with. One is called AREMA methodology. Stands for auto regressive integrated moving average. It's again, in the world of statistics. It's somewhat straightforward and simple. That's a methodology I've used to actually forecast website data, just pure page views, sessions whatever it is. And it's fairly accurate. it'll get the actuals within usually about plus or minus 5% depending on certain factors. I would honestly like ask the AI as well, it will train you on how to do these things.

Sometimes just uploading your data and say, "Hey,look at the data I uploaded." Tell chat GPT or Claude, whatever. A little bit about it, the relationships. What do you [00:33:00] think is the best statistical methodology to use here, and can you apply that statistical methodology to what I uploaded? We never really touched on regression. There's, I think, eight or nine different flavors of regression that you can apply. This is another statistical technique.

It's something that's used very common quite often. The AI will probably tell you based on what I'm seeing and what you've told me I would apply this methodology. Ask it to explain it to you, and then ask it to apply it to that dataset and do the testing as well. So I think really just understanding what you've got.

Let AI train you up on it. It's the easiest thing I can tell people to do is just go ask. It's quite good. It'll tell you. I think the biggest thing is just be familiar with your data. Know your data in and out. Understand the definitions of the metrics. Try to record everything as best you can.

One of the biggest things we run into especially with digital UTM codes, they've been around for before I got into the space. I think they started in 2002 before Google Analytics and any of that. Just tracking those types of things and coming up with uniform tracking systems and naming [00:34:00] conventions will help immensely. It's staying organized and tagging everything properly.

But, ultimately, like rely on AI. That's why it's here. It's supposed to be a crutch. It's supposed to help you out, do these things. And so again, without nerding up the end of this, know your data, utilize AI if you have the right privacy permissions in place and you're confident data is secure. Upload it into AI and just ask it.

I do. Not all the time cause I know a lot of this stuff, but it helps me do this type of stuff faster, and more accurately. So, I know we don't have to do too much forecasting in our current day-to-day but, when I was in my old agency, this is what we use. It was just automated. Granted, I had data scientists underneath me that were a little bit more hardcore and up to date on actual stuff. But use AI, use your system. That's what it's there for.

Erik Martinez: I think that's a great piece of advice. Fantastic stuff, man. This is not our typical, podcast. This is probably a little nerdier and geekier than people are expecting, but it's a hugely important component of all the things we do as [00:35:00] marketers and business people trying to figure out, " Hey. Money coming in, money going out. Are we making a profit? How do we grow our businesses?" And this is some of the glue that sits under the hood to make sure that we can do that.

This was awesome. And thanks for bringing me back to my old statistics class and professor, whatever his name was.

Pat Barry: It's not, the most exciting topic, but, statistics and math can predict the future. That's what I try to tell people. It's your time machine, if you will, for numbers in the future. But, lean on AI. Honestly, that's why it's here.

A lot of the stuff I described, I used to use Python coding. I still do, but fortunately, Claude writes the code for me. A lot of the times I just sit there and edit. So happy to answer any of the questions. I'd love to come back for another podcast.

Erik Martinez: Absolutely man! Anytime we can geek out on other types of AI conversations too. Well, hey, Pat, thank you again. I really appreciate you coming on.

Folks, that's it for today's episode of the Digital Velocity Podcast. I'm Erik Martinez. You [00:36:00] guys have a fantastic day. Thanks.

[00:27:00] Thank you for listening. If you have enjoyed our show today, please tell a friend, leave us a review, and subscribe on your favorite podcast platform. Visit the Digital Velocity Podcast website to send us your questions and topic suggestions. Be sure to join us again on the Digital Velocity Podcast.

Scroll to Top