How to Use Multivariate Creative Testing to Instantly Improve Creative Ad Performance

One of the fun things about being a creative is, well, you get to be creative!

But, what works – and why? What if you had data – actual numbers!! – that mapped out the best possible creative solutions?

Thanks to Dan Pantelo, CEO of Marpipe for taking us behind the scenes for a look at how his company, Marpipe, magically transforms creative by taking elemental pieces, fitting them together, testing variable combinations, and magically forming the answer [with data] to what’s the best ad to run? Get in line to test Marpipe.

When you automate creativity, you discover the best options. Marpipe is the first platform that enables multivariate testing for ad creative – empowering your creative performance with data. Small changes in creative lead to big changes in performance. With the ability to create controlled creative variations, marketers can continuously optimize top performing ads by understanding the underlying features that make audiences respond. Source: Marpipe

How to Use Multivariate Creative Testing to Instantly Improve Creative Ad Performance, an interview with Dan Pantelo, CEO of Marpipe

BARBARA

Can you tell us a little bit about Marpipe and the future of marketing as you see it, Dan?

DAN

Yeah, definitely. Marpipe is a multivariate creative testing platform.

We believe Marpipe is the first truly, and completely, data-driven way to approach creative.

We kind of see creative and marketing right now at a really major inflection point in the industry. I think that’s the place in the industry where we’re going to see by far the most disruptive innovation happening in the 20s decades.

When you trace some of the most monumental changes that have happened in marketing so far, you see three different trillion-dollar waves that happened in terms of disruption and marketing activity.

When the internet first came along, there was a big problem with consolidating inventory. Everyone had a different website. In the early internet, if people wanted to advertise, they had to contact the website owner and create deals to advertise.

When major advertising networks and DSPs came along, they consolidated all the inventory. Shoppers who wanted to buy ad inventory could just sit in one place and do everything they wanted.

This created a trillion-dollar industry.

And then, what happened was the next problem in marketing, which was a problem solved in this past decade: audience targeting.

Now we had all this inventory online, and we knew where people’s eyeballs were. But, we didn’t know much about the people we were targeting.

We had a whole host of technology providers who came along and solved problems in audience identity resolution. This helped us serve our ads to the people who were most interested in seeing them.

It’s a much more efficient advertising ecosystem for both advertisers and consumers. A lot of these companies ended up being acquired by Facebook and Google and rolled up.

This solves the audience identity problem, which was one of the major disruptions in the past decade.

To answer your question, what’s the future of marketing going to look like? I can’t predict that far out.

But I can certainly predict that in the near-term, creative is the next big problem to solve.

If you look at CMO speakers on major industry panels you’ll see everyone’s talking about creative, creative as an important problem to solve.

Because the smallest changes in creative have some of the most profound impacts on performance, right?

Like we can change the ad background color from blue to green. Or, we replace, a cactus with a palm tree. Every change is going to have a profound impact on performance, either positive or negative.

If every single, tiny visual stimuli detail can vastly impact performance, then we need to ask: how is it that we make all these creative decisions?

What information do we use to decide what our creative should look like?

This is something that’s stayed the same since the Mad Men era. The approach to deciding what our creative should look like is something that is mostly guided by arbitrary taste-making.

That’s not necessarily a bad thing. It’s just to underscore and highlight that it’s not a data-driven process. It’s not procedural or systematic.

When creatives make creative, the typical process has been, and is still today, they are interested in making something that works well. They present it to the client, the client either likes it or doesn’t like it and gives feedback.

And then we essentially arrive at something that’s a mix between what the creatives like and what the client likes, right? And the creatives are pitching and selling the creative to the client, who’s hiring,

BARBARA

And as a creative, I know how that goes. You ask the client, what are you looking for? Then, you back with what you came up with. And you get feedback. Would you agree that sometimes the client overlays their preferences on what the creative is versus what the actual audience might engage with most?

DAN

It’s not sometimes, it’s all the time.

BARBARA

Can you talk a little bit about the different levels and the layers of measurement that you do? For example, one of the Marpipe videos I saw on LinkedIn shows multivariate testing in action, if you will. Showing a kitten does well, but then there’s an abstract that also performs well for other variables.

DAN

Essentially, what we try to do is eliminate arbitrary feedback in the creative decision-making process. This is one of biggest day to day pain points for creatives today.

To do that, we need to understand not just which ads work best, but why they work best and exactly what visual factors are driving them to perform the most.

The same thing happens when you see a person and you instantly know whether you’re attracted to them or not within milliseconds.

It’s just a human instinct.

You make the same judgments when you see a piece of creative, and it’s a combination of the stimuli. But certain factors are more important to you that win you over and some that just don’t mesh with you the right way.

And so what we try to do is understand exactly what those factors are. That’s what multivariate testing can enable.

For instance, if we run two ads and the only difference is that they have different fonts; we’re isolating that variable. If we run them in the same circumstances, and there’s a difference in performance, whichever one performs better, it’s because of the font.

When it comes to actual creative deployments, there are so many variables that are important to measure: positioning, colors, copy, font, facial expressions, the people, demographics, the product, and more.

Marpipe allows you to enable proper multivariate testing to facilitate tagging, and categorizing or more than 500 elements. We then make groups of creative components. For example, let’s take background colors. How many of you want to test? How many logo options? Then, we come up with the number of ads we need to test.

BARBARA

When you think about how much time it would take an in-house creative department to put that all together, that’s a head-spinner.

DAN

You’re correct. And the craziest thing, that just blows me away every day, is how many people who are doing this manually.

BARBARA

Wow! How does Marpipe make so many ads? Do do you use AI?

DAN

Although there are AI components, I wouldn’t want to generalize with a buzzword. But there are some AI components to what we do.

For instance, when you upload some creative or creative assets into our software, it’s automatically tagged by machine vision. We have a machine that looks at whatever you upload and tells you exactly what it is tagging.

We can reference it and attribute performance to really specific tags later. The actual analytics also rely on AI. But to make to automate the creation of all the ads is not AI. That’s essentially a configuration of what’s called a batch render farm.

BARBARA

A Batch Render farm – what is that?

DAN

If we were to mass create something in the same way that it might be created on an assembly line. It’s kind of like an equation, like a formula, right?

You essentially templatized a certain piece of creative and then you pull apart all the different elements. So, you divide it into different groups. And then you make more versions of the elements, and then they’re all combined and recombined together in every possible version.

The batch render farm automates the act of doing all of those things. When you’re using Marpipe and you’re using our ad builder, you can build 36 ads In 30 seconds. It’s incredible. You’re creating on an exponential level, and you almost don’t realize that it feels like you’re just making one ad.

BARBARA

Wow. Marpipe is like creative magic! You mentioned that there are some pain points for creatives earlier on in our conversation. So what would those pain points be? And then what benefits does Marpipe bring to them?

DAN

I love talking to my friends in the industry, especially in the agency game about this. It’s just a lot of fun. Because really, there’s a lot of friction with different stakeholders in the marketing process.

And, it’s not a productive friction. Some productive frictions should exist. These are not. Creatives want to create the thing that will perform the best, right? Of course, it’s better for everyone. The creators feel fulfillment from what the product delivers. The company that hired them is way more willing to invest more in their creative production. And, so, everyone wins.

BARBARA

Exactly right.

DAN

But, creatives don’t understand what works or why. Because they’re not the ones running the ads and looking at the analytics, those are the paid media people, the media buyers, right?

BARBARA

Well, that’s true. Can you stop a minute and tell us who is at the table? There’s a lot of players in this process.

DAN

The branding folks come to enforce a set of rules. The metrics-drive organic social folks have a deep understanding of what their audiences engage with most naturally on organic mediums.

But, what we see all the time is people assuming what works best on organic social also means it will work on paid or vice versa. And we never see a correlation between the two, which is interesting.

When we talk about what kind of content we should make for organic social and for ads, these are two separate conversations that should be informed by two separate data sets.

We should look at what works the best across both channels and mash those insights together. We’ve always seen different things work well on different channels on different mediums.

BARBARA

So much to consider and would you agree that graphics are key to marketing on the web? If you don’t have great graphics, are people going to pass you by?

DAN

I agree. Which is why making the creative investment today is super important.

At Marpipe we have a full-time illustrator on our team, at a technology company, which is something people might raise an eyebrow at. Especially when comparing us to other technology companies at the same stage.

But for us, it’s, it’s essential because we need to communicate properly through the digital mediums that our users are on.

BARBARA

Right now there’s a waitlist to join Marpipe. Can you tell us about the background?

DAN

I started running a performance marketing agency called Pantelo. We specialized in running paid media campaigns and acquisition, performance marketing, or for startups, venture-backed startups. These are typically folks who had raised a pretty good amount of money from venture capitalists and such.

And it was our job to manage their campaigns, to make their creative and run it.

We were under a performance mandate, which meant the only thing our clients cared about was results. How much money did we spend with you plus your fee? How much purchasing did our ads generate? ROI was the top-line metric for most of our customers.

Getting arbitrary creative feedback from our clients was frustrating. We had a chip on our shoulder because we knew these things would work well. But if the client didn’t like how the ads looked, it was back to the drawing board.

We knew, though, that the most important thing that speaks volumes to a customer is performance. If one ad performs the best and you don’t like how it looks as a client, are you going to still run it? 99 times out of a 100, there’s a 99% chance that you will.

It doesn’t matter if you like it, or I like it, what matters most is that the customers we want the most like it.

To find the best creative solutions, we were looking for a multivariate testing system. As an agency, we wanted to make data-driven decisions.

We wanted to justify creative decisions rather than try to convince the client that it looked great. We were looking for an out-of-the-box solution. We couldn’t find any – it didn’t exist. And so we started doing it by hand ourselves. And it was like, it’s just impossible.

BARBARA

I can’t imagine doing that. Creating all those ads must have been very time-intensive.

DAN

It was like a never-ending slog. We didn’t like that it was so time-intensive, and required nailing the methodology and not screwing up. We got to the point of asking: is this even worth doing?

Because of the insane level of investment, most companies would decide that it’s not worth it. We believed there should be tools to automate this process. A lot of these interactions are like rote manual processes. Was it possible to automate them? So, I took the agency profit and hired some engineers to start building these tools for us to use internally. And when we had these tools, and we used them internally, we would have an edge over other agencies.

BARBARA

That was smart.

DAN

So we started building these tools, and we went down a lot of dead ends. We screwed up a lot. We used our client base as guinea pigs with their consent, of course. And, a lot of things that we tried didn’t quite work out. But we ended up, after tinkering with it a lot, nailing a really solid process. But it was all like back end raw code built by engineers. No one knew how we were doing it. Clients began passing over their creative assets in a particular format. And they knew they’d get back hundreds of ads right away.

BARBARA

That’s impressive.

DAN

They thought we had a big dark room somewhere in a Manhattan basement filled with creatives.

And for the first time we were able to tell these brands here’s why your ads work. Here’s what people love most about your creative or never use this version of your logo again. I know you think it looks the best, but people hate it.

We call this creative intelligence. There’s going to be a new world of creative insight where we’re going to learn how people react to creative components. And, that’s cool. Multivariate testing is the only way to unlock this data.

Some sophisticated brands spend over a million dollars a month on Facebook ads. What happens when their creative works for the first time? We will see an inflection point that changes everything.

When word got out about our testing, we started getting inbound calls. People weren’t interested in working with an agency, they just wanted to know how we did multivariate testing.

And that’s when we knew we were a technology company and not an agency. That’s when we realized we created something that filled a company vacuum. And so, that was exciting.

We decided to take the leap and showed a few investors. We ended up raising our first outside capital in q4 of last year. We conducted a fundraising round from some of the best investors in the space. When they saw the tools we built, they were impressed and they saw the potential impact Marpipe could have on the industry.

Funding allowed us to make the leap and pivot to taking the product from the engineers into a self-serve system anyone can use. Our engineers were the only ones who could have used the product that they built. This is a bad business model. You never want your engineers to be service providers. You want engineers to focus on building more products.

BARBARA

Congrats on cracking the code! What’s next for Marpipe?

DAN

I see a world where the best creative solution is no longer about who has the most convincing and articulate pitch in the room. But, instead we say, hey, let’s run it through Marpipe. And let’s just put data behind every hypothesis. Now that you have all the answers, you no longer have to guess and shoot in the dark.

Performance increases make brands happy. And, one of the things that makes us most excited is empowering creators with data for the first time. Now creatives can speak the same language at the table with the paid media folks and the CMOS. They can show what performs the best. And they can say if you don’t like our looks, great. We can change it. But this is what’s going to perform the best. And so that empowers them in a completely new way at the creative table. So we hope to see that shift happen, and we’re happy to empower it.

BARBARA

How can people get in touch with you, and how can they sign up for your product?

DAN

We essentially stopped taking managed service customers when we decided to start building a self-serve product, accessible to all as a subscription service. That’s what we’ve built.

Even while we were building it, we were collecting a waitlist. Our waitlist is now over 500 people. These are folks from all areas of the space from agencies to big brands, to small startups. It’s a diverse mixed group. The best way to get into the product is to sign up for the waitlist. Frankly, we have more demand than we thought we would. If you want priority access to skip the line, so to speak, there’s an option to enter your billing info and pay $1.00.

BARBARA

Thanks for the fantastic interview, Dan! Congrats on all of your success and cheers to the future of creativity!!

About Dan Pantelo

When Dan discovered martech, he became obsessed with the ability to digitally automate the process of selling. Now he focuses on building these tools with a scrappy and visionary team of marketing and engineering experts. He can’t imagine having a job that’s more fun than this.