Michaela Dempsey (00:00):

Nicole, thank you for inviting us to participate in the CLOC Solution Lab series. Super excited today to explore AI-powered automation and how it can elevate the practice of contract management. Next slide. I'm Michaela Dempsey, Chief Marketing Officer at Evisort, and thrilled to be joined today by Chau Nguyen, Business Program Manager at Microsoft. Chau manages a set of contracting technology portfolios within the legal operations team at Microsoft and has been in the contracting space for the past seven years. Chau is passionate about automation. She has built a career on looking for repeatable work that can be eliminated and simplified by discovering technology that enables automation of that work so humans can focus on higher value work, and the things that they love. So let's go to the next slide.

(00:49):

What are we up to today on this webinar? Well, first we'll kick things off by looking at legal technology trends that we're seeing in the industry. Then a deep dive into Microsoft's AI story and learn how they leverage machine learning for their legal operations. And finally, we close out with a live Q&A with Chau. So let's get started. Next slide. On the road to CLOC's annual Vegas event, my first time with CLOC, we certainly saw a lot of new surrounding legal technology. And while at CLOC, the numbers were dropped on all of us. Legal technology investments are certainly on the rise. The survey shared that organizations are getting value out of their investments. 92% of the departments that adopted new technology have achieved better ROI from automation than they actually expected they would see. Seeing this kind of results, over half the legal ops teams, or 54%, indicate that it's a high priority for them to implement a new technology solution. But here's where it gets really pear-shaped.

(01:53):

When it comes to the type of technology, only 22% of legal ops teams are making use of data science or AI technology. That's 78% of companies, of people, who are using, or still using, human intervention to solve legal ops needs. Now, this is understandable, as AI used to need a data scientist directly to run models and figure things out. But those days are behind us. And we're going to get into that a little more with Chau. Next slide, please. I also wanted to highlight a comment from Jenn McCarron, CLOC board member and director of legal operations at Netflix. I was recently listening to her CLOC Talk podcast episode on CLM technology, and it was a really interesting discussion about where the industry's going and what teams should expect as they look to adopt some of these new capabilities.

(02:44):

One thing that stuck out to me was Jenn's call out, that no matter what, with this new technologies, people's jobs will be changing. And that scares people, and you shouldn't be scared because change isn't always bad. Change is usually really, really good. As she says, this sort of transformation work doesn't happen overnight. It takes time for people to adopt new tools and new processes. So I'm curious, let's go to the next slide, in this group, let's do a quick poll. How many of you have used AI in your legal operations? Chau, I'd love to get your thoughts. What do you think? Now we heard up above with the other stats that 78% of people weren't using that, but where do you think where that survey might fall?

Chau Nguyen (03:30):

Well, I hope everyone's going to pick the first one, but I have a feeling it maybe the second option.

Michaela Dempsey (03:37):

Yeah. I think it's going to fall in the middle, a yes. Or we have a plan for the year. I hope it's not the fourth option, but we'll see soon enough.

Speaker 3 (03:51):

So, we've got 21%, yes, and it has proven effective. 30%, yes, but we have not fully utilized it. 40%, no, but I plan to in the next year. And 7%, no, and I have plans to do so.

Michaela Dempsey (04:06):

Well, it looks like this group really is making waves with AI or on the road, so that's awesome. So with that, let's roll into Microsoft's AI story. Thank you all for participating in the poll, and Chau, can you start off by sharing about what legal ops looks like at Microsoft?

Chau Nguyen (04:25):

Yeah, absolutely. First of all, I'd like to share the mission in the legal ops department, and that is to really support the legal and compliance piece for Microsoft. But one of our primary goals is to be the low-cost alternative for legal work. And we do that using the people, the process and the technology. There's three strategic pillars that we use in operation, and those are, the first one is the building of the solutions. The second part is to really operate them at scale. And the third part is to sustain those solutions, to make sure that we're able to support the volume of the business and the size of our company. Next slide, please.

(05:10):

Our AI journey started about five years ago. So we had the opportunity to partner with Microsoft Research in order to prove out this concept of machine learning and AI in the legal domain. This team consisted of some of the best data scientists in the world. The result was proven successful. But one of the challenges that we faced was that it was very time and resource intensive to be able to build and train our own AI, and not to mention even the initial investment that you need to make to do this work. And so, the barrier to entry is quite high and difficult for a lot of people. The next slide, please.

Michaela Dempsey (05:50):

Well, quick question. When did you first realize that AI was a possibility or even a necessity? Going back to that other, well, you can talk about it here, but I was thinking in that other slide.

Chau Nguyen (06:06):

Well, so AI made it... We hear a lot about that in the media, machine learning and AI, even with Tesla being the prime example, but really, our team, my director right now, had the opportunity to go to AI school. And I think that's when the whole thing started, was we're able to explore this possibility and have the opportunity with the Microsoft Research team. But that's really the backstory of how we even looked at AI as a possibility.

Michaela Dempsey (06:43):

Okay. Sounds great.

Chau Nguyen (06:48):

So fast forward to today, you can see that there's now a vibrant marketplace with plenty of AI providers, Evisort being one of them, and a business bond model that people can really just go out and license without having to make this huge initial investment, meaning that you have to have a team of world renowned data scientists, and investing thousands of man hours into the project. Not to mention, you needed to have tens of thousands of sample data to even build out a model. You can just do that today by essentially writing a check and have a company like Evisort coming in to help with that. The nice thing that I like about where we've progressed with AI, is even an attorney or a legal professional today can just go in and trade the model with a high level of accuracy, and really not having that many samples to feed into the machine in order for it to learn and operate. Next slide.

Michaela Dempsey (07:46):

That's interesting.

Chau Nguyen (07:48):

Oh, sure.

Michaela Dempsey (07:48):

Wait, let's go back. You mentioned with the finance group needed extra, or when we talked before, that they needed to extract contract intelligence. So I was curious, because I know when you first started, you were really looking at a lot of your procurement contracts. Was it accounts payable, procurement, financial, finance analysis? What was the initial driver when you started looking, or you and your team started looking at Evisort?

Chau Nguyen (08:25):

I can go into that a little bit more in more details in one of the later slides, but the initial problem we were trying to solve for was how do we quickly look at a set of contracts? And I think at the time, we were looking at maybe 40 to 50,000 contracts. How do we quickly do that and really identify some of the nonstandard terms in the payment terms within the supplier contracts, so that we can go out and negotiate or renegotiate with suppliers so that we could save the company money? But that's really the initial project that started all of this and made us go and start looking for AI providers in the contracting space.

Michaela Dempsey (09:03):

Got it. And I know you've been with us for a while. When did you kick off that search?

Chau Nguyen (09:10):

It was [inaudible 00:09:12] memory recalls. I think it was in 2020.

Michaela Dempsey (09:17):

2020, okay.

Chau Nguyen (09:20):

Yeah. Yeah. I believe.

Michaela Dempsey (09:21):

Was it pressure from the pandemic or was it sort of January-ish?

Chau Nguyen (09:28):

I think it was around the pandemic time, but I know that wasn't the reason why.

Michaela Dempsey (09:33):

Got it. Okay. Sorry.

Chau Nguyen (09:35):

No worries. Okay. Next slide, please. And so I wanted to talk a little bit about our journey on how we selected Evisort to be our contracting partner. We actually looked at some of the first party solutions, the custom models within Microsoft. We also had a couple of other providers, Evisort being one of them. And it turns out Evisort performed very well compared to even our own internal first party solution. I don't know if there's other colleagues of mine on here, but Evisort turned out to be the company that really perform well for us. And the other thing that I like about Evisort, is you guys have this robust level API so that we can integrate into our existing complicated infrastructure. When you're dealing with millions of documents, it's not very efficient to manually upload those into the Evisort platform, and so having those APIs will really help us migrate them over in a programmatic way. So those were the selling points for us, and that's what, really, our business needed at the time.

Michaela Dempsey (10:46):

Right. And I think you're also using the API for some of your Power BI dashboards, yes?

Chau Nguyen (10:52):

Yeah. So the reporting piece is definitely something that we needed to integrate. APIs we use for training a massive amount of contracts, but also to pull the data and servicing into a reportable format, and leveraging Power BI to do so. So you're absolutely correct, Michaela.

Michaela Dempsey (11:13):

Yeah. Yeah. And just in the flexibility, overall, I think you guys have really done a wonderful job showing the broad range of how to use the product. It's pretty exciting, overall, from what I've seen when I've had the conversations with you guys. So what are some of the projects that you're looking at now?

Chau Nguyen (11:43):

Well, that brings me into... Let's move to the next slide and I can talk about some of those how do we leverage Evisort in the vendor management space, and then the next slide, I'll talk about how do we do that overall with a broader organization. But I mentioned earlier, payment terms is one of the clauses, or you can think of like a provision that we're looking inside of these contracts, supplier contracts, to identify what is nonstandard and how can we go and renegotiate those with the suppliers to get a more favorable terms for Microsoft. But we also had other groups in our legal space that come to us because they wanted to check out the privacy clauses and some of the security commitments we have in these supplier contracts. So there's a collaboration between a procurement group and a legal group to investigate some of these other clauses that are inside those contracts.

Michaela Dempsey (12:39):

So were these all related to related projects or separate initiatives? And in general, how many contracts are you talking about?

Chau Nguyen (12:52):

Yes, they're separate projects, which made it interesting because each group has their own business initiative and what they're looking to solve. The volume, depending on the group, but we're looking at between 50,000 to a 100,000 sets of contracts, primarily the masters, that they're looking to extract those [inaudible 00:13:17].

Michaela Dempsey (13:17):

Right. And when you're building those things out in general, by using automation, what has that allowed you to do for these teams?

Chau Nguyen (13:30):

It allowed us to quickly give them the results that they needed, so that they can build out essentially a risk profile for these suppliers, and even having actionable insights based on the results. The other thing that's nice about this is they don't have to go and employ a bunch of humans and then train them to do this work, because that could take a lot of time, a lot of resources, where they can just quickly do this with our team.

Michaela Dempsey (13:58):

Right. And at this particular time, it was 2020, so time was more of an imperative. When you talk about doing it quickly, what is the frame of quickly?

Chau Nguyen (14:10):

So we looked at the... We did an estimate, of course, because we didn't go out and employ people. But I think of one of the projects we had for the 100,000 contracts, it was going to take about five to six humans, vendors that we had to onboard, and we had to train them and look at these contracts over a period of six months, versus, I think it took like a week for us to train the model, have somebody help us validate that the model is performing correctly and then giving an output. And this is all happening over the course of a week versus six months. Did I lose you guys?

Michaela Dempsey (14:58):

Yeah, I think so. If you could go back, you ended, "Training the model and performing it," then we dropped there, so we didn't get the [inaudible 00:15:09].

Chau Nguyen (15:10):

Sorry about that. Not sure what [inaudible 00:15:11].

Michaela Dempsey (15:11):

We were all waiting, like a cliffhanger. It was a big cliffhanger for all of us.

Chau Nguyen (15:18):

So we estimated if we didn't have this model in place, it would've taken us six months to even get the results that we were looking for versus a week's worth of time and effort.

Michaela Dempsey (15:29):

Yeah. Yeah. No, that's amazing. That is amazing. And the accuracy. So generally speaking, it's more accurate. What we've seen is that our system is about 95% accurate, but most AI is more accurate. Humans, I think, come in at 60 to 80% accuracy. Are you seeing those same types of numbers?

Chau Nguyen (15:55):

Not specific, in terms of those numbers, but what we've noticed that is that when you introduce the machine learning factor, yes, you're getting quicker results, but you also have to look to see how much do you want to accept that risk that there's no human eyes on something. We didn't specifically measure the accuracy for this particular payment terms extraction, but there's other projects, and that's on the next slide. I'll talk about where we extract metadata from contracts. And that proved to be highly accurate when we compare that with the human being involved.

Michaela Dempsey (16:34):

Well, that's great. Well, let's go into that. That sounds pretty exciting.

Chau Nguyen (16:39):

Yeah, so we talk about, okay, how do we leverage AI in the broader organization? Or I'm going to talk about that right now. In the operational excellence space, our team has to... our team is responsible for a set of non revenue contracts. I think we're up at one and a half million contracts now in the repository. So data hygiene is critical for us. And when I came in, we had to swap from one CLM provider to another. So you can imagine the data hygiene is not all that accurate when you migrate from multiple systems over time.

(17:17):

And it was brought to our attention that some of these contracts needed some data hygiene cleanup, and in order to do that work, we couldn't just... We don't have unlimited resources and time, and so we had to really use... figure out a way to identify those sets of problematic contracts. And we leverage Evisort to do that work. So once we've identified those problematic contracts, then we can figure out, okay, how do we extract the correct data using AI? And then essentially, just correcting those incorrect metadata. On the broader [inaudible 00:17:59]... Go ahead.

Michaela Dempsey (18:01):

No, no, that's amazing. And so, when you're correcting the metadata, what steps go into that overall?

Chau Nguyen (18:14):

Gotcha. So the first thing we had to do is figure out, okay, out of those one and a half million contracts, what is important to us that we need to focus on? And so identifying the subset of contracts that we need to fix, and then going and extracting the correct data, comparing it against the current set of metadata inside of those contracts, we have human do a data sampling of... Is Evisort accurate enough for us to use the extracted data versus what's already in the system? And we've determined that, yes, we can use Evisort to override those problematic data that we had existing, where the humans were entering them initially. And so with that, we just overwrote all of the existing data with Evisort data. So we did a replacement of them.

Michaela Dempsey (19:02):

That's nice. Yeah. We did a webinar last month with a customer, and they were explaining that they're working on a M&A project and they brought in a third party consulting firm and they had found 34 critical documents. But when it ran through the system in five minutes, there were actually 50 critical documents. And this particular company does not have the volume of contracts, so five minutes is where they were. I know it's not five minutes for everything, but it is interesting because it does show that as good as we are as humans, we miss things.

Chau Nguyen (19:45):

That's right.

Michaela Dempsey (19:46):

We don't want to, it's not like we're trying to, we are just human.

Chau Nguyen (19:50):

That's right. When you're dealing with large volume, you're going to end up missing out on some stuff, but yes, it is interesting that you mentioned the M&A space because that's the next thing I was going to talk about, is how do we leverage AI in the corporate transactions, particularly in the M&A space. For us, we needed to quickly respond to the regulators and also conduct due diligence on the deal, and really understanding the contractual obligations to move the deal forward. And so, business velocity is really important to us. Even though we're in the early stages right now of using Evisort in the M&A space, we've seen very promising results. And we're able to engage with outside council to train the model., and we've found some really interesting results because of that.

Michaela Dempsey (20:40):

Yeah. No, we found our customers really are doing a lot with M&A, but not even a... It could be a more internal, it wouldn't necessarily be M&A, but similar reasons of why you'd be looking at contracts, to either clean them up, et cetera. So it is quite successful in that realm. And then the other area that we've seen people really super succeed is in ESG initiatives. So really looking at key clauses, because there aren't standard clauses for... Let's say you're looking for environmental parts or pieces, there isn't a standard clause at the moment that speaks to that. So being able to train and look at that with our... We have a new product that's coming out in July, Automation Hub, that will allow you to really find all these things really so easily and so quickly. I know we were talking about earlier before the call, but I can't wait to get you guys on it. It's going to be really awesome.

Chau Nguyen (21:51):

Yeah, absolutely. I'm very excited about that. And then you can see there's a lot of possibilities out there that we haven't explored with AI. These are just some of the examples we're looking at now, but I think in the next couple of years, there's going to be a lot of other use cases that we can apply AI to.

Michaela Dempsey (22:06):

So what's the benefit, overall, for the organization, have you seen, over the projects that you've done over the last couple years using Evisort, or any of these types of projects? What are the benefits overall?

Chau Nguyen (22:23):

I mean, the benefit is going back to being... We don't need to hire a lot of humans to do this repetitive work. We can leverage AI for that, and we can quickly get the results without having to wait for months. Now we've reduced that from months to weeks, and it's going to be todays when we get the model more robust and we can train it more quickly.

Michaela Dempsey (22:49):

Right. And that kind of ties back to when we were introducing you, really freeing things up. So it's not that Microsoft is looking to eliminate humans. It's eliminating humans for tasks that are not productive, overall, that you can use a system for, and allowing the employees of Microsoft to then be elevated with the research that's in their hands a lot faster so that they can make better business decisions overall, right? That's presumably the goal?

Chau Nguyen (23:29):

That's right. We're definitely not trying to eliminate jobs with AI. What we're doing here is freeing up time so that people, like you said, can perform higher value work because there's a lot of work out there, and we can't even scratch a surface with what we can do with AI and how we can free up time for people. Because you can think of a paralegal, it's not very cost... It doesn't make sense for them to go in and comb through these contracts, looking for repetitive terms, where we can get some machine to do that with. And then the person can go and focus on, "Okay, now that we've identified this is a non-standard clause in this negotiation, and how do I go and help facilitate that negotiation and work with the business?"

Michaela Dempsey (24:16):

And there's another piece that I think is a really, really important factor. I'm older, no one can see me right now, but I am older, and so I grew up where you're kind of used to doing traditional non-automated tasks and you sort of just assume them. But the thing is that as... It's silly. And the thing is that you look at what was in the news, a great resignation. You're bringing in people into this new work environment and they need tools that are going to really help elevate them, help them grow their career and eliminate what really should be eliminated manual processes, because of technology. We have a customer care initiatives that they hired a person right out of college, and that person's never done manual work because they've been using the product. They don't even understand why people would do that, but of course they wouldn't, because if you have a solution or there's technology available to lighten that up, then you're allowing people to do the job that they dreamed of, right?

Chau Nguyen (25:39):

That's right.

Michaela Dempsey (25:39):

So then you're excited to go to work, because no one dreams of running through contracts, no one dreams of living in Excel sheets and looking for errors or doing parts and pieces there. So, when you look at it that way, and you look at the fact that it's finally... It's like a flying car. It's finally allowing you to do what you wanted to do in the first place.

Chau Nguyen (26:04):

Yeah. That's right. I couldn't agree more.

Michaela Dempsey (26:10):

Awesome.

Chau Nguyen (26:13):

Okay. I have one last slide and then I'll hand it over to you, Michaela.

Michaela Dempsey (26:19):

Okay.

Chau Nguyen (26:22):

So what did we learn out of this journey? Some of the stuff that we looked at... I mean, if I were to go back, this is what I would do, or I would advise people to do, is the first, test out your potential technology partners, meaning that bring your business use cases to them and let them help you figure out a proof of concept and see if that works to solve your business case before you select that partner. We've done that with, I believe, Evisort and another provider that I mentioned earlier, plus our first party custom solution. And then make an informed decision based on the outcome. The other thing is really align your legal and business priorities. Meaning that, for us, we want it to look at more of the nuanced languages inside of those contracts, and metadata extraction is secondary to us.

(27:09):

And so really look for a solution that would fit your... that would align with your priorities. You're not going to find one that's going to solve every single thing, but choose the one that was going to solve for most of your problems. And the third thing, is really consider the culture of the organization. We have some attorneys that are open to this technology, but there's still others that are not open to it. And it's an uphill battle with AI, really. So change management is really important, and anticipate that not everyone's going to adopt to this and it is going to take time for people to be accepting of this technology.

Michaela Dempsey (27:54):

Yeah. I agree with you. Change management is always the death of everything, whether it be your new year's resolutions or the things you're doing at work. Making change is difficult. I actually just finished a book called Atomic Habits, which I recommend to everybody. It's great because it gives you a way to do little 1% changes. And so this might be something that will help your organization at Microsoft. I would also add in on the test, at Evisort, I know that our team always says, "Don't trust us, test us." And then because I'm in marketing, I'm always like, "Don't say don't trust us." But what I do say is... I agree with you, test it out, test out actually the solution you're going to do. Have them in front of you show that how they're pouring the contracts in and see it find what you're looking for, so that you really get a true perspective of what you're looking at overall.

(29:00):

Well, thank you so much for sharing that. And let's go on to the next slide. Your journey's been really awesome, and it's been great working with you. And again, I can't wait to get you to see the product that's getting released next month. So excited for that. And I wanted to share a little bit more about Evisort overall for anyone who may be hearing about it for the first time. So Evisort offers an AI powered contract lifecycle management and contract intelligent platform.

(29:30):

As Chau shared, we specifically built the platform so that anyone can use it. You don't need to be a data scientist and you don't even need to know how to code. We've already trained Evisort with over 10 million contracts so that it understands the context and the meaning of legal language. The platform itself is also built to be easily integrated into your existing document repository, so there's no need to migrate data. Evisort can pull straight from your Google Drive, your box, your Dropbox, or even digitize your old paper contracts.

‍