Lauren Magee (00:00:06):

Thank you everyone today for joining us. My name is Lauren Magee and I am the content and education manager here at CLOC, and we're thrilled to bring you today's Ask the Expert panel exploring 2023 tech trends for legal ops with Civix and Gainsight, brought to you by today's sponsor of Evisort. Before we get started, I just want to point out a couple of things to help you get the most out of today's session. On the platform you'll see at the bottom that we have a chat function. Please use this to chat with each other, put in any thoughts that you may have during today's presentation.

(00:00:42):

For any questions that you may have for the speakers, please put those in the Q&A box. We did take questions before this presentation today, so they're going to spend a lot of time answering those questions, but we'll definitely have some time at the end for some live questions. And then also we do have a session evaluation after the session ends, it'll just pop up in your browser, so we'd love if you to take just a quick moment to complete the survey about your experience on today's session, and also we're recording today's session as well, so you'll be able to rewatch it. We'll send you the link following the session. All right, and with that, I want to again welcome you all and also thank Evisort for bringing us this presentation today. And I'm going to hand it over to today's moderator, Hal Marcus.

Hal Marcus (00:01:33):

Thank you Lauren, and thanks everyone for being here today. It's a pleasure to lead a session like this. Today's going to be really a conversation, much more than a presentation. I've led many webinars and speaking sessions over the last three decades at this point in the legal tech space, and something I learned a long time ago is it makes my life a lot easier when you just invite intelligent, conversational people into the room and then just ask them questions. That's like the easiest job in the world, and so I'm happy to have that very easy job today.

(00:02:06):

Very briefly, my background. I am the VP of product marketing at Evisort. I joined the company just about two months ago. Previously, I've been accompanied like Lexus Nexus, Thompson Reuters, OpenText, RecoMind, most recently DocuSign. I started my career as a practicing litigator at a large Wall Street firm in my native New York, but I moved out to San Francisco after a few years and joined Lexus Nexus, got into a career in legal tech and I have been doing that ever since.

(00:02:34):

In one of those roles at a small to medium sized software company. For five years I doubled as general counsel. So while I was doing lots of other things for the organization, I had to negotiate every contract for every customer, every partner, everything that we ever did in the org. Had zero tech to support me to do it. So it's natural that I've gravitated toward contract management solutions and AI for legal use cases that would've made me much more powerful as a solo attorney in that organization as our GC. So that's my career path. That's how I came to be here. I'm now going to introduce our colleagues and let them tell their backgrounds and I'll encourage them to do that thoroughly so that you know where they're coming from and what expertise they're bringing. So since I have to put people on the spot now, I'll start with my colleague Colby Mangonon from Evisort, our AGC. Colby. Can you give us your background?

Colby Mangonon (00:03:30):

Of course. Hi, I'm Colby. I'm the current AGC at Evisort. Previously, I've been at companies like SOCi and Citrix. I have a pretty heavy background in SaaS and commercial contracts. I have helped build out commercial legal teams, worked a lot with Legal Ops, bringing on legal tech and building out the process and really just working to make things more efficient in our day-to-day lives. In every position, I've always found it's really important to look at all of the stakeholders like sales and make sure that these implementations really work for everyone that is involved and that it's important too. So that's always been a big focus for me is not just bringing on the tech, but making sure that it works, that you have a return on investment and that the implementation goes really smoothly.

Hal Marcus (00:04:21):

Great. Thanks Colby. And Brittany, let hand it off to you.

Brittany Leonard (00:04:25):

Hey everyone, I am Brittany Leonard, General Council for Civix. This is a software and service company and I have been for about eight years. Prior to that, I have law firm experience as well. And I think for me, Legal Ops is huge because I know back then when I was first starting out, like Hal said, it was not prevalent, it was not a thing, and I've just seen the value and the tremendous speed that it gives to people to not only get the work done but get the work done efficiently. And there's just so many benefits that people need to take advantage of with the tech. And then also I do content creation on the side. Many of you know that about me and I see the value in it just exploding through everything. I know we're going to talk about ChatGPT and everything that it gives you the benefit to give you, my biggest thing is about career happiness and work-life balance, and I think with the tech and if people embrace it, those two things will become a lot more prevalent in people's lives.

Hal Marcus (00:05:32):

Great, thanks Brittany. And Caitlin, over to you.

Caitlin Foster (00:05:36):

Hi, I'm Caitlin. I'm Senior Corporate Council at Gainsight. We're a really small team, so I'm responsible for negotiating any and all agreements that find their the way to me. I'm currently working on our CLM implementation and just anything else that kind of falls into our lap. Before this, I negotiated aerospace contracts at Lockheed Martin and TV programming contracts at DISH. So I have a wide array of responsibility or experience in terms of subject matter and the size of the teams that I've worked on, the organizations that I've worked in, and actually this is my first CLM. In about a decade of practicing the law, I spent all of that time negotiating contracts in Word and saving them on servers. And so now we're upgrading.

Hal Marcus (00:06:36):

It's a big leap forward when you go from... Well back and forth, email attachments with Red lines has been around for so long and it's still done so much, but it makes such a difference when you implement an actual system around it, not just links in a spreadsheet or a SharePoint server, but we will get to all of that. Thanks for being here, Caitlin. Let me just change one little note that Lauren made earlier where she was saying at the end we'll have time for Q&A and that's not untrue, but please enter your questions as we go. Like I was saying, this is really a conversation today. We have all the questions that have come in advance from the people that are attending and we're looking to give some guidance on what they'd like us to cover. So we do have some stuff to start us out, but enter your questions as you go.

(00:07:28):

We'll address them on the fly as best we can. I have very little idea where this conversation's going to go. I do have three categories that are things that we're going to focus on. The first is legal tech trends just in general. So we'll start off fairly broad, see where that takes us. Then we'll talk a little more specifically about legal tech for your org. In other words, the criteria you use and why are you implementing the tech, what are the things you're looking for? How do you assess them, how do you implement them? Well, those types of topics. We'll see where that takes us. But we do want to reserve time to talk a little more pointedly about AI powered CLM, the role of AI in contract lifecycle management, the goals of contract lifecycle management, best practices from implementing, Caitlin we'll learn from you what's your experience with this first time ever now using a CLM and what a difference it's made.

(00:08:17):

So keep the questions coming in as we go and we'll try to keep this really lively and moving. So that having been said, we're going to start off very broadly. One of the questions that came in was from someone who said there are a new general council in the tech industry. So people obviously in this particular call can closely relate. What would you say are the top three areas of interest to stay apprised of the most this year or just in general, but particularly this year? What are we seeing happening now? What's really hot on your plate? So again, since I have to put somebody on the spot first, I'm going to do that once again to my esteemed colleague because she can't get mad at me. So Colby, what's high on your list right now for trends and focus areas?

Colby Mangonon (00:09:04):

Of course, I think data privacy is always top of mind. I think we've seen that there is a trend that data privacy is going to be more important, that with these integrations that we're looking at with legal tech stacks, we really have to be thoughtful about what our vendor's data privacy looks like, what their InfoSec structure looks like and how that's integrating with our own customer data, particularly if we're in a highly regulated industry. And I do think that the introduction of OpenAI and how that integrates with the data privacy, I think that's really important. It's something that we should always be apprised of. There's more and more tech coming, it's not going to go away. So I think we have to be really thoughtful about what our liability is in that area, how it integrates with our system, what our procurement function is to kind of catch and gate keep any of our concerns or liability from our vendors.

Hal Marcus (00:10:03):

I think OpenAI is going to just pop up in our conversation a few times today.

Colby Mangonon (00:10:07):

Just a couple.

Hal Marcus (00:10:10):

Caitlin, let's go to you. What's some top areas for you?

Caitlin Foster (00:10:13):

I think within privacy, I'm really interested right now in whether we're going to see federal legislation to kind of put a stop to the myriad state legislation that we're seeing or if state legislation is going to proliferate. I think the intersection of privacy and employment law is really interesting too. Data retention versus not keeping your data too long. And then just the constantly changing barometer of what are our standards, our negotiation standards around privacy, particularly limitation of liability caps. How are other people thinking about that? How should we be thinking about that?

Hal Marcus (00:11:01):

Let's explore that one just a little bit. And I'm always curious genuinely about this, about how the conversation gets started. How do you know what are becoming industry standards and what others are doing with their contracting terms?

Caitlin Foster (00:11:16):

I think a lot of that is talking to your network, seeing what other people in the industry are doing, and also what are you seeing in your deals, what are people asking for the most frequently? Where are you landing the most frequently?

Hal Marcus (00:11:33):

And Brittany, let's move on to you with top trends, but we're going to circle back on that one. I think as we talk more about contracting, that's important. But Brittany, what are your top trends that you're noticing this year?

Brittany Leonard (00:11:44):

Yeah, so I think top trends, especially in the tech, you always have to think of your intellectual property and that's a huge one right now. Obviously with the privacy and the IP, with things like AI and ChatGPT, there is no regulations on it. I hear from saying, "we ban all of our lawyers from using it," or a company will say, "we have blocked that website," what have you, but what is going to be the legislation that comes out to say how you use it, what can you use?

(00:12:17):

I know someone I spoke with when I was at Legalweek this week said, "they're allowed to use it in their company, but they have to tell people which portion of their memo that they wrote or whatever they're writing, which portion they used ChatGPT from." So it's very interesting to see the take on not only a privacy issue and like Caitlin said, like an employment issue of where's that fine line of how you're using it to make your work more efficient, but also how are you still protecting from a human perspective? How are you protecting your employees and how are you protecting the IP that your company creates?

Hal Marcus (00:13:05):

So it's very interesting. I'm hearing in all of these focus areas, and I don't by any means disagree. In fact, this has been a hot button topic for me as Colby knows well, but I'm hearing a lot about protecting your organization from the use of certain kinds of tech, the data privacy implications, what are the regulatory standards? Here's what I haven't heard yet and I'm going to raise it because it came in one of the questions and I thought it was very interesting, the implications we're seeing right now from cost cutting and macroeconomic conditions, what implications does that have? What's the impact we're seeing of that on legal technology? Are we seeing more of a stress of taking advantage of tech to reduce costs? Are we seeing a reluctance to invest in tech because of all the cost cutting? And the question that came in actually took it in a little different direction too, of how do you overcome that? If there is that reluctance, how do you get the organization to understand the value of what you're trying to implement from a financial standpoint?

Brittany Leonard (00:14:05):

I think people are so afraid to implement that because they're afraid, oh my god, my job's going to be taken away, what have you. My job's going to be reduced. But the thing that you have to learn is it's an assistance. AI is an assistance, you still need that human check on it. So something that would take you, like me and Colby and Caitlin will review contracts if we need to find a certain provision, if we had to do it manually, we're looking at two to three weeks. AI, we can do it in a day or two. So we still have to have that human element to it, but it significantly reduces the time. It doesn't reduce the human job of it.

Colby Mangonon (00:14:46):

I think we're also seeing people use the current tech stack that they have to its maximum capability. I think that's one of the things that we're seeing is that there's more growth within a current account versus the cost of onboarding something new. But I also do think that if you can focus on the efficiency and kind of do the homework on the front end of how this can make everyone's life easier, get the stakeholders to really buy into why this is a great investment, then you can find whatever technology tool that fits those needs the best. And when you actually implement it, it feels really good. It's making you more efficient. I think it's more about being thoughtful about having these tech tools. I think a couple years ago it was like, cool, this tool does X, Y and Z. Let's bring it on board.

(00:15:36):

This one does this. Let's bring it on board. Our tech stack is 47 different vendors and it writes our emails for us, it makes our calendars. And now we're looking at what's the real value in all of these different technologies? How necessary is it for us? You may be in an industry where not having to do your own calendar, it makes a big difference. And so that's something that is really important to you. But you may be in an industry where that's an easy one to say, "Hey, let's look at our procurement list and say these vendors got to go because we have to save budget to expand our CLM and get more value out of that.

Caitlin Foster (00:16:14):

And I think there can be resistance too around time. Just we're all busy, our teams are busy, how are we going to find the time to implement this? That can be really overwhelming for a large legal department where you've already got a ton of data, it can be overwhelming for a small legal department where you're one or two people and you have a ton of stuff on your desk coming from the entire organization. But the sooner you do it, the better it's going to be. Like if you start now, you're not going to have a pile up of more data that you're going to have to retroactively make fit into a system. You're going to have the processes in place, so how much easier are you making your future life and how much better are you making your future organization by just doing it just starting, even if you can't do it perfectly from the start.

Brittany Leonard (00:17:07):

Someone said to me the other day, they're like, well, if someone's resistant, I'm going to show all the people that aren't resistant so that the resistant people just come on board. I'm like, no, no, no. What you do for the resistant people is figure out their issue and show them specifically to them how the tech will help solve their issue. Don't just try to slash people in together be like, oh, these people like it, so just come on board. No help the people that are resistant by coming to them and showing them how it can make their life specifically easier.

Hal Marcus (00:17:43):

I think that's a really important point for adoption is sort of the walk them through it, the hands on, as relying on FOMO in this particular industry tend to work all that well. I think we've seen that a lot over the years for sure. A lot of our conversation is trending and understandably, and by the way, that's fine for Evisort towards CLM and the value of AI in that space. But I want to be mindful that we've had some questions and one just came in and we had some beforehand about other sort of key areas of tech that are really important in the current time. Some that we were asked about are things like remote proceedings, info security related tech as it pertains to legal, keeping an eye on that or being responsive to breaches perhaps and notifications and that kind of thing. And I'll just throw it open. Anything else that is sort of a high profile area in any of your experiences or legal tech outside of CLM that is high ROI getting a lot of attention right now.

Colby Mangonon (00:18:46):

Yeah, I think in the HR world that's a really high data privacy concern. So if you have any kind of payroll tech or you have something that has your employee data because that's PII, so it has a higher level of requirements and regulation that are placed on that data. And to Caitlin's point, looking at data retention, well now we're seeing laws like the California Pay Equity Act where it's asking us to retain employment data for a longer period of time. So now we're having to balance what our data retention policies are under data privacy and what our other regulatory requirements are.

(00:19:27):

I think that any legal tech has value, any tech stack has value depending on what your organization needs it for. But there always is a balance of is it better that we do this? What is the security risk? What is the data privacy risk? Things like HR, I mean it's pretty well accepted that you are going to have a payroll system. I don't think a lot of people are doing it by hand still, but I think that those are kind of the thought processes that you have to go through when you're looking at InfoSec. Is it better to have somebody who is a contract InfoSec provider or is more of a tech stack or do you want to bring somebody in-house that can do it? So everything is kept within your protected environment.

Hal Marcus (00:20:14):

Caitlin, Brittany, any thoughts on other sort of key focus areas outside of CLM that are getting a lot of attention in your orgs right now or in general?

Caitlin Foster (00:20:23):

Not in my org. I mean, I've known people who are implementing document retention systems, especially if you've got more litigation in play than we do. But definitely hearing about that one. And I think too, some of it is how can we use other systems in legal? We're leveraging Salesforce more than historically we have and incorporating our tech with our Salesforce instance. So it's kind of more just how can you integrate the stack?

Brittany Leonard (00:20:56):

Exactly. Just like you were saying, integrating like we used monday.com a lot. So things like that. How do these things work together to just make it seamless almost?

Hal Marcus (00:21:06):

That's been a hot button topic. I'll just throw in as a quick thought from our perspective at Evisort because a large part of what we're about is applying AI to extract the contract data, convert the pros of contracts into meaningful actionable data points. So then the next question becomes what can you do with all of that? Certainly search, run reports, dashboards, mine, find answers to questions, but is that something that's entirely within the world of legal ops or can that be delivered out to other systems so that business users across Procurement, HR, Sales and other departments are getting the value of all that information? So increasingly we're focused on extensibility and delivering those contract data points into a range of other systems. So I think that speaks nicely to what you're talking about of taking advantage of the rest of the tech stack of the org.

(00:21:56):

So having done that little soapbox moment, let me ask a question on that about the role of legal ops then. Do you see that translating, and this relates to one of the key questions that came in too earlier. Do you see the role of legal ops being looked at a little bit differently these days? I mean, by the way, I have enough gray hairs that I know Legal Ops is still a relatively recent role period, but I remember when no one knew what that meant. But in these recent years, have we seen it shift away from purely making Legal more efficient and have we seen it shift to more of an overall business mindset and tying in with the tech stack and the rest of the operations around the org? What's your experience with that?

Caitlin Foster (00:22:40):

I don't think that shift is just Legal Ops. I think that shift is Legal. Legal moving to be a member of the business team instead of in its silo. I mean everybody's heard the old saw about the deal prevention team and it's our job not to be the deal prevention team. We need to make the business successful just like the sales people do. So we need to be problem solvers. And I think that goes for ops and it goes for the people negotiating deals and it's all of us.

Hal Marcus (00:23:10):

So instead of the deal prevention team, you want to be the deal facilitation team.

Caitlin Foster (00:23:14):

And if you can be an ally and convince your colleagues that you're an ally, it makes your job better and more enjoyable and you're going to get better results across the board.

Hal Marcus (00:23:25):

And for what it's worth, that's definitely a trend that I've seen over the last, I mean quite a few number of years now. It's gradual, but I think it is hitting sort of an inflection point these days. It's a much more common theme.

Brittany Leonard (00:23:40):

Oh you're saying earlier how the resistance, so long people have been resistant to change to doing things differently. And I think the more that people, I guess you could say hop on board or what have you, the easier they see that their life becomes. And like Caitlin said Legal or Ops, call ops, there's so many vendors out, I was telling Colby the other day, there's so many vendors out that I haven't even heard of that can... From translations to investigations to e-discovery, there's so many things that Legal just doesn't have to use on their own that the whole business can integrate with to make it more efficient that why wouldn't you use it? Why wouldn't you want to make your life easier, make the job easier? And then obviously that increases revenue generation. So it's just like it's an overall benefit to everybody in the organization. For sure.

Colby Mangonon (00:24:42):

We have seen on teams where Legal is the most willing to integrate or to bring on tech stack where then Legal Ops is kind of that point of contact for other parts of the business to come in and say, "Hey, how is this working for you guys? How can we leverage this?" They see that somebody else has brought it on board and been the brave one to have a vendor to do these things and they go to the Legal Ops professional and say, "what's integration like? What's the implementation like? How can we get in on this? Because it seems like it's working really well for you. So I do think at times Legal Ops is a bridge to the business and to just maybe the CFO or somebody else who wants to get in on the tech that legal was the first to implement.

Hal Marcus (00:25:26):

That is a big change. If you're reflecting just based on my experience and we're definitely-

Brittany Leonard (00:25:34):

Instead of calling us a cost center, you can call us a revenue center.

Hal Marcus (00:25:40):

I was just going to say we've seem to have hit a nerve with some of the comments that are calling in of like, yes, please stop. Do everything you can to undermine this notion of Legal as a cost center. It should be a revenue driver, should be a business facilitation, it should be all of those things. And I've seen it be those things. I just think that the attorneys that I've worked with over the years and in-house teams that are those kind of innovative drivers have been the exception more than the norm. And I think we're gradually seeing that start to become the norm and even that level is impressive. That touches on something. I want to raise that. It was another question that came in earlier and right before we started, I mentioned this to all of you that a question came in about law firms versus in-house council and their differing views of the technology and the use of legal tech.

(00:26:29):

And I shared with everyone that it made me laugh because my experience back at a firm quite some years ago was we had zero tech until in-house council came to visit and sort of went, "why don't you have this and why don't you have that?" And then the next day we had that tech, we moved very quickly at the behest of in-house council, and yet I hear from your comments earlier, it sounds like sometimes you'll be hearing now from the firms, oh we've got this great new thing, we're using this cool thing. And then sometimes you're a little skeptical about what does that mean? What's the value of that? Why does that make me more interested in working with you? Caitlin, I think you had raised that. Can you elaborate a little bit on that rift?

Caitlin Foster (00:27:10):

Yeah, right after all of the news broke about ChatGPT, an article came out about some huge multinational law firm that was like, we're using AI and here's the AI that we're using. But they didn't really say what they were doing with it. In response, we had a lot of internal push like, hey legal, why aren't you doing that thing? But we don't know what they're doing with it. We don't know how they're engaging it. And so there's a lot yet to be seen I think in terms of the value that it's adding and just how it's being used beyond a marketing point.

Hal Marcus (00:27:54):

Yeah, this is obviously the hot topic right now in AI and I think we can pretty soon, we should pivot. We have a lot of questions about CLM specific implementation evaluation, so we will get to all of that. Let's maybe use this as our pivot point for that because it's obviously top of mind for everyone. And so let me just briefly lay a foundation for those that, you probably are all aware, but for anyone attending that is not that aware with recent changes, GPT-3 was released by OpenAI almost as an afterthought. They weren't planning as I understand it's release until they got to GPT-4, which was released like a week ago and they did not plan to release anything until then. And then they sort of got the sense that an escalation was happening and they didn't want to be left behind.

(00:28:40):

So apologies to anyone at OpenAI if I'm misrepresenting, but this is my understanding of how it came to pass. They released GPT-3 and within I think two months if I'm getting these stats, it had 30 million users and 5 million people using it daily. So I mean an extraordinary take up of that tech right from the outset. And then within about three months they released 4, which was really what they figured was going to be the minimum viable product, if you will, as we call it in our space. And now Google has released Bard, Microsoft implemented this into Bing. So there have been fantastical stories about experiences that people playing with this have had that are scary and weird, and there have been magnificent stories about what it can deliver and anyone that hasn't played with it, I encourage you to, it's really kind of mind blowing how much it can replicate the human communication pattern.

(00:29:43):

I'm not even here right now. I am GPT-4. I should have said that at the beginning. My name is Hal, you should have known. Anyway, that's a dated reference, but some people don't know it. So with that foundation, this is obviously very significant for this space. We don't know exactly what its long-term implications will be at Evisort we're working with it, others are as well.

(00:30:06):

But before we get into anything too specific on CLM or such, what are your thoughts generally, what are you seeing out there? What are you hearing, Caitlin? Obviously you've heard from outside council, we're using it, we think this is great. Are they doing that to tell you we're going to keep our costs low because we're doing say research and core writing faster? What do you think is driving that? Are they just trying to show that they're... Somebody else got the Hal reference. Thank you. But what is behind that? What's driving that? What do you think? Caitlin, when you heard from outside council that they were saying we're using this great tech, do you think that's what's driving it? Why they want you to know that?

Caitlin Foster (00:30:49):

That wasn't our outside council, that was a law firm that kind of just came out to the public and was like, Hey, we're doing it. We're the first right. And so I don't know what, I think, yeah, they're probably trying to drive buzz around what they're doing. Maybe they're trying to create an implication that there's going to be efficiency driven by this. But it would be pure speculation for me to comment.

Brittany Leonard (00:31:21):

I think it's trendy and I think I was really hesitant to even look at it for weeks. And then I think it was a week and a half ago, I asked my friend, I was like, "how do I log into this thing and explore it?" And he's like, "you really want to do this?" And so I think it's something like I said, that it's going to help people with getting more efficient in their work. But then there's the other side of it, and I'm not one to shy away from both sides. There's the other side that's very scary that can be used detrimentally. And I was talking to someone yesterday up in New York and he said he was an attorney for an insurance company and people were using ChatGPT to basically figure out the amount of money that they should give somebody for a claim.

(00:32:05):

And he was giving a reference of a three year old that broke her arm and they're like, oh, ChatGPT only thinks you should get this amount of money. And I'm like, oh my God, imagine your child and something like that and you are now relying on AI to give you an amount of money that a human should be looking at and should be verifying. And so there's many ways that it can be abused, but there has to be some type of wraparound of how to use it efficiently but don't abuse the system either. And right now it's just so trendy, like Caitlin was saying that we have to get some type of parameter around it. But what is that we don't know yet.

Colby Mangonon (00:32:48):

I think there's definitely responsible and irresponsible uses of it and I think that lends itself better to certain industries than others. It's a large language model. I think for legal in some instances it's incredibly useful as a tool. Mostly because if we're looking at drafting, there are standard clauses that we draft from, there are playbooks that we have internally, we have our clause library. Things that we can pull from for this large language model to be useful. I think just with any other tech vendor, you have to look at the specifics of how it's being used. I think a lot of times my concern is where you're looking at the large language model just being used on its own as opposed to somebody who's integrating it into the use of their platform because then you're looking at more parameters.

(00:33:40):

You can ask about what's the data retention policy. I think we just have to look at some of these red flags and understand what we need to be concerned with when we bring on something that has the OpenAI, it's like is it GPT-3? Is it GPT-4? What LLM model? What large language model are they using? Is it OpenAI?

Hal Marcus (00:34:00):

And there are others coming.

Colby Mangonon (00:34:02):

There are a number of vendors yes that provide it.

Hal Marcus (00:34:06):

It's almost become synonymous like Kleenex now with GPT. But that is really just one model.

Colby Mangonon (00:34:11):

Yes.

Hal Marcus (00:34:11):

And so one system,

Colby Mangonon (00:34:13):

It's looking at what system they're using, what the retention policy is, if they're using the large language model. So if you're logging into ChatGPT and you're dropping information in, I don't know what the data privacy restrictions are there. I don't know what they're doing with that data. I don't know if they're retaining it, if it's being used to train the model additionally. But I think when you bring on AI as an integration into a system, then that system is responsible for determining what the retention policies are, if it's being trained on that data and looking at the parameters of that. So I think that's something that every in-house team should be looking at and kind of evaluating when they bring on tools that have this AI. Because I'm sure there are options that are not great, but I do think that it's a really incredible powerful tool and I don't think that we should be scared of it. I think we just have to be thoughtful in how we integrate and implement it into our business use.

Caitlin Foster (00:35:13):

And in terms of the tools that you're talking about, the clause libraries and the things that are truly help aids instead of work replacements. I think about all of the years that I spent negotiating my contracts in Word and saving them on a server. And there were times that I would marvel at how much easier it made my job than somebody who did their job on a typewriter or even earlier than that. Find and replace is a magical function if you have to change out a defined term throughout a document or find a needle in a haystack in your document. And so I would think about that, wow, how much easier is my job now than it would've been 30 years ago? And I think if you think another 20 or 30 years into the future, you're going to have another young attorney saying, wow, how did anybody ever practice the law without this? It just makes my life so much better.

Hal Marcus (00:36:11):

And I think with the case of LLMs, it's going to be, you had to draft from scratch? You had a blank page and had to come up with every word? You couldn't just give it a few parameters and have it produce something that was not very good but got you started and then you could edit and refine? How did you live like that? That's I think what we're likely to be hearing in a few years.

Colby Mangonon (00:36:35):

And I think from a legal standpoint, especially in the CLM space, I'll speak for Evisort, our intention is not to replace the attorney, it's to provide tools to make their lives easier, to make them more efficient. We always intend that the attorney will be a supervisor. So we think of it as if you have commercial counsel under you and they are drafting something, you are going to supervise them. You're going to review what they're drafting. You may be escalated up to give approvals for something they can send back. It's a very similar situation where it's a tool that you are then reviewing and determining, yes, I want to give this to my opposing counsel. It's not really like we're pressing a button and our large language model is sending an email out to opposing counsel with being an attorney. So I think there is a responsible use of it. I think there always should be a supervisory aspect from an attorney's standpoint because that's an ethical issue, which is outside of all of the other issues that we're looking at with these large language models,

Hal Marcus (00:37:37):

We've had a bunch of comments come in on this and one of them in particular touched on something that I think is a really important distinction, especially for those that aren't well versed in this tech and what's been happening. And you touched on it earlier, Colby as well. There's really two pieces to this. The LLMs, these large language models, which is the category for which ChatGPT and Google's Bard and others that are emerging are related to, they're about language recognition. Basically. It's an extreme version of what we're all used to when we're texting and it's suggesting the next word for us. And then of course it corrects you inadvertently with the wrong one and you get annoyed. But that's the same kind of tech just at a much more advanced scale. And what's really impressive about it is how fast it's advancing now.

(00:38:26):

And we're seeing that that improvement become sort of exponential. Might plateau at some point, I mean I'm in San Francisco, we have Waymo self-driving vehicles with all this worrying things on all cars all over the place, but we're years past the point when we thought there weren't going to be drivers and there were still drivers. So some things do take longer and don't go as well as we think, but right now we're seeing a massive improvement from one level to the next and it looks like it's getting exponentially better. And because it's so good at replicating human communication and it's trained on the enormity of the internet, it can give you this sense that it's really thinking, that it's really giving you solid information back. And that's the issue. There's two sides to this. One is that it replicates communication so well and it's very valuable for that. But the other piece is that the information that it's basing all of its writing on is not necessarily the information you want to rely on. So we can be fully wrong.

(00:39:30):

Somebody else had a comment about biases that come in politically and in other areas because when you train on everything on Twitter, you got a good range of physicians and it doesn't really have the capacity to decide this one versus that one other than based on probability. So that's a really important aspect in terms of the ethical and reliable use of this for legal. Colby, please.

Colby Mangonon (00:39:56):

I think that's why I think it's really important for us to look at the parameters, like I said, so I'll speak for Evisort in particular, but when we use the large language model, when we're talking about what we're developing and what we're looking at for the use of LLM in our product, we have our own proprietary AI. And that AI is really great at looking at a contract and determining this is a jurisdiction clause, this is a venue clause, this is a limitation of liability clause. And so it provides those parameters. And then even further, you have a playbook or a clause library that you've created in your own internal instance that has your fallback positions. It has essentially what your playbook is.

(00:40:39):

And we can use the LLM, give it parameters to base on what our proprietary AI is determining is a venue clause. It can look at the fallback positions that exist within the clause library and then it can use those to generate something. And that's where the generative AI, you really need to look at what the parameters are and how it's being controlled. Because the large language model on itself, of course it's going to have these biases and things that aren't maybe as useful in the legal field, but there are uses where you can design it to work more efficiently and better for the particular use case that you want it for.

Caitlin Foster (00:41:20):

And I liked what you said Hal about it can look like it's making decisions, but you've also got all of this other stuff coming in and you don't know what it's using as the basis. My job is not just spitting out language and using a set of rules for fallback positions. I am making discretionary choices all day every day based on the business needs and the client, the terms of this agreement. And I take a lot of factors into account to determine how I'm going to respond to something. So you are always going to need, not just from an ethical standpoint, but just from a business standpoint, a good attorney monitoring your software.

Hal Marcus (00:42:05):

For sure. And this is probably a great way to delve a little deeper into some of the more practical questions that have been coming in about actually implementing AI powered CLM. What are some of the best practices? How has that gone for people? And Colby, I think this touches on an important point you made. The foundational AI that is turning your contracts into data is now something that I think with... It's like GPT is sort of raising the ceiling of what people are anticipating AI can do. But that doesn't necessarily mean that that's going to be the primary implementation, though we love it and we're using it for appropriate things in our world, but at the core of it has to be that reliable data of what's in your contracts and knowing that that's being used to infuse and inform the decisions that Caitlin's talking about.

(00:42:56):

So with that in mind, let's talk about that experience. What has worked well for you, Caitlin and Brittany? I'll ask you to, since Colby is biased, as am I, but what's been your experience with seeing the AI, what it's brought to bear in the organization as you're working with your contracts? And by the way, with that in mind, what are the things you were looking for when this was implemented? What are the criteria that people should be looking for? Because I'm getting a lot of questions on that front.

Brittany Leonard (00:43:29):

For me, the intelligence behind it is just so profound that things I would've never thought that it could pull up. The fact that it can pull up alternative language to a search that I do that has similar language to that to say, Hey, I know you didn't search for this, but maybe you're thinking of this language as well is just so profound. And like Colby said, putting parameters around it is really efficient and really helpful. And then obviously you have people CLOC that come out and say, Hey, here's out there and have these type of webinars to show you here's the benefits of it, but use it in a way that's still protecting your organization. And I think that's huge.

Caitlin Foster (00:44:16):

Yeah, I think for us we were looking for something that could track a ton of fields. In our ideal world, you can make the entire contract analyzable data and reportable data and you can see some of that coming to bear. We are still in the process of optimizing what we're doing, but definitely, I mean we want to be able to see where all of our liability caps are. How high are we going on multiples, on dollars? Are we staying within our guidance 90% of the time or 10% of the time? We know that now because they're basically two people negotiating all of our contracts, but we won't always know that. And so it's a great time to set that baseline. We also want to be able to analyze all of that data and report to our board, to our CEO. We want our finance department to be able to get in there and find the data that they need regarding payment terms, whatever that may be, notice information.

(00:45:29):

So all of that I think played into our decision. And then of course we wanted an interface that would allow us to work collaboratively on deal negotiation so that it wasn't just, hey, legal, here's a document and now I'm the gatekeeper for all the information. The AE can initiate the ticket, he can see where it is. Other people can get in there and work with me or at the same time I can call people into that document. So all of those were really big factors for us in terms of the functionality we were looking for.

Colby Mangonon (00:46:03):

Something I think it's really important when you're looking to implement a CLM or really any tech is to do an audit before you even start looking, look at your process and say what is working? What isn't working? Because if you have a certain e-signature platform that your entire organization is obsessed with and they want to make sure they can continue to use it, make sure that's an integration when you look at your CLM. If you have something that's really not working and that's the pain point, make sure that's solved by your CLM. I think a lot of times people, they come in not really knowing what they're looking for or what their needs are. And I think the implementation is so much smoother when you know what the pain points are and what you don't want to break, because then when you bring it on board you can say, Hey, we don't need to change the entire process.

(00:46:52):

We only need to change these three things to make it more efficient and to have people on board. Then when the implementation happens, I think it needs to be something that has a feedback loop. It can't just be a set, this is a training, you have one training, it's a document, you're taking it. I mean legal operations is where I have worked with incredible legal operations professionals who took on that role of bringing in the feedback of understanding, hey, I've gotten 17 questions about how they take it from review to sign. So I know that either my team or I need to create training on this. Let's do individualized training with each team. So it's a living, breathing process and implementation is going to be different at every single organization because there are going to be things that there are resistance to or that people are willing to take on. So I think you just have to understand that it'll take a while and you need to let it be that feedback process as you bring it on and doing that foundational work first will make it easier to implement it.

Hal Marcus (00:48:00):

And let me double click on that real fast. And then open this to everyone because one of the questions that came in was the comfort with email legal attachments, like I said earlier on, we've just been doing that for so long now. It's just such a comfortable typical... So how do you overcome that? I mean Caitlin, you were talking about it being very collaborative in that being an advantage. Was there a resistance on that? Did people understand that pretty quickly?

Caitlin Foster (00:48:24):

Well, we're still in the process of working all of this out. I think it's going to be a hit once it takes, but certainly it's going to be an adjustment. I don't think we'll see resistance. I mean, we're in a great position as a SaaS company. People want to adopt new technology. I think there are some companies that you would need more resistance, but there's definitely going to need a learning curve. And I completely agree with Colby about feedback, the feedback loop. And I would also say really leverage the team at your CLM because they know that tech better than you do. And I can say we went pretty far down the implementation path before we realized that some of our underlying assumptions were just wrong or conceptualizing thinking about things incorrectly. And we had divided work in a way that, oh, actually these two pieces talk to each other. So you can't really defy that work in that way. And so how do we run that?

Hal Marcus (00:49:28):

Sorry, what's an example of something? So the division of labor, I was looking for an example of what you were saying of something you assumed and it turned out not to be quite right about the process.

Caitlin Foster (00:49:37):

Yes. So originally we had one person implementing workflows and one person implementing the backend data stuff and document retention and all of that. But what we didn't really understand was that the workflow intake form talked to the data fields on the backend. So we really needed to be co-developing that. And I think had we been able to ask better questions and run that plan by the Evisort team, instead of being like, this is what we're doing, really try to get them to sort of black hat the plan, maybe we could have saved a little time there.

Hal Marcus (00:50:22):

And Brittany, you're dealing with a considerably larger organization and legal team. What's been your experience with that in terms of adoption and changing people's behaviors and such?

Brittany Leonard (00:50:33):

Yeah, I think it's a huge thing to get people to want to change, especially in the legal field. I think me and Colby talk about this a lot. Law firms are still hesitant to change where it's more flexible and more willing to change inside in-house teams, especially in a tech company because we do tech. So it's a new tech, it's trendy, it's fun. People are more adept in wanting to see new trends in a tech company obviously. So it's more about getting people on board to say "it's going to make your life easier," than saying, "here's a new tech that we have, learn how to use it." It's like, I don't want you to go out in a boat without a paddle or canoe, whatever you want to call it.

(00:51:25):

I want you to have the tools that you need to learn how to use it. And I think that's where you get people that are resistant to this is to say, Hey, this is what it does. But don't worry, I'm not going to just let you go out there and try to figure it out. We're going to do training. I'm going to show you how it works. I'm going to show you what you need and if you have questions or if you want to know something specific, let's talk about it. Let's discuss it because we want to make sure everyone feels comfortable with it.

Hal Marcus (00:51:55):

In our remaining time. We're getting to that point here where I'm looking through what haven't we covered from the questions coming in. I just jotted down four things I want to make sure we cover and they all are acronym based, so let me rattle them off and then we'll try to make sure we do the lightning round here and cover all of these. One is about RFPs and use of RFPs versus consultancies or both. What's a good approach to start, just so I'll give you the heads up of what's coming. Another is regarding the importance of APIs. There was a question earlier about good practices around NDAs and making those easier and faster. And finally, ROI, making sure you show ROI and how to do that effectively. So, let me start with the RFP's. RFP process versus consultancies versus other methodologies for evaluating and assessing CLM providers. What's your experience? What's a good approach? And feel free to expand that to experience with other products over the years.

Brittany Leonard (00:52:54):

I think RFP versus consultancy, I think it depends on the people that you have. If you're going to do an RFP and then you're going to review all those proposals in-house, that's great if you have the bandwidth, if you have the resources. Consultancy is great if you don't have that. I'm very hands on so even if I have either one, I want to be able to see the benefits, the cons, I'm going to have questions. So for me, an RFP versus hiring a company is, do you have the funds to hire a consulting company to look at all that or do you have the bandwidth to look at RFP?

(00:53:33):

Or do you want to hire a few temp agents? Whatever you want to do. You just have to look at the cost efficiency. Someone had mentioned that in the Q&A. Cost efficiency is huge in-house teams. So do you have the bandwidth to hire a consultant firm or do you want to bring that in-house, but knowing if you bring it in-house, I guess your ability to look at all proposals may not be as thorough as a consulting company that you hire to do that on its own.

Caitlin Foster (00:54:05):

And in a smaller company, I think a lot of that is also driven by size because we are small enough that neither one of those options really would've made sense for us. We have a two attorney, four total people legal team, and so our process was a lot more informal where we just kind of pulled our networks about what they used and if they liked it or not, reached out to those companies and kind of self-managed getting demos and making a decision. It was just so much more casual. But again, that's just because we're small.

Hal Marcus (00:54:42):

Apologies the mini sounds there, they just said we have five minutes left. But that's a perfect driver. Yeah, definitely. Obviously it's going to be affected by the size of the org and what tools and what means you have in place. As you look at the criteria, and I think this is also going to be very much a function of the size of the organization, but as you look at those criteria, how important is extensibility with APIs, which you would use then to integrate potentially the workflows, setting up tickets, delivering data out to any number of workflow solutions of ERP solutions, of sales solutions like Salesforce, CRMs. There's so many ways to be tying into contracts with other parts of the org. What's been your experience with that, Brittany? I see you're nodding. I think you've done a little work on that front.

Brittany Leonard (00:55:34):

Yeah, are you talking about tying other departments into drawing them on the CLM that we-

Hal Marcus (00:55:42):

Yeah, building on the APIs to integrate from your CLM out to other solutions or tools?

Brittany Leonard (00:55:49):

Yeah, I mean for me it's huge to get our IT department involved. I need them because they're going to be the ones to know all of the vendors across the board that we use, all of the third parties. I am sure there's tech that some departments have that I didn't even think about. So how can we integrate all that? How can we get IT on board to sit there and be able to fit everything together seamlessly? And I think that's a huge thing. And they want to be involved, especially on the security forefront. So why not having the people that want to be involved, be involved as much as they can be.

Colby Mangonon (00:56:28):

And I think it also speaks to just what's important to your organization. Like I said, what's not broken? So if Salesforce is the way that you want to create a ticket for your CLM, I think most legal vendors or tech vendors are going to have those capabilities of those main integrations. If there's one that's different that you use that's not necessarily out of the box, then it's just something you have to speak to your vendors about. But I think it's very common to use APIs with tech.

Hal Marcus (00:56:59):

Great. All right. Lightning round here. So I'm going to keep plowing through ROI. How do you show ROI and to whom do you show it?

Brittany Leonard (00:57:07):

I mean, I'm going to speak for all three of us on here. Being able to sit there and say, Hey, we only take two days to review. The other side's taken three weeks. I think it's a huge thing to be able to say whenever someone says, oh, legal takes forever. Actually, no, we don't. The other side does. So I think that's a huge one. It's huge benefit for us to be able to say it's being done efficiently, but there's a thousand more that I can think of. I know you guys have some others.

Caitlin Foster (00:57:43):

Yeah, I think allowing people to take ownership, access the data themselves, and also our ability to respond to the CEO or CFO really quickly when they have a question about the terms that we've been entering it to generally. All of that I think shows the entire organization ROI on the product.

Colby Mangonon (00:58:03):

And Evisort does have dashboarding capabilities and things that allow you to show, it takes us X amount of days to finish an NDA to get it through the system. But my personal favorite ROI, like Brittany was mentioning, is when sales is really happy with us and our winning channel is going off and we're like, Hey, I was on that deal and we're getting shouted out saying we negotiated these things and we're getting it done. I think it's really important that that's something that people are happy about.

Brittany Leonard (00:58:34):

We're in an M&A transaction right now, and I needed to find all of our contracts with change of control languages, strong assignments, change of control, and if I would've had to do that with all the contracts we had for that specific entity, it would've probably taken me a week, two weeks. I was able to do it in about an hour. So I think that's huge to be able to take an M&A deal and speed it up just simply by pulling out what you need from your contracts.

Hal Marcus (00:58:59):

That's fantastic. I'm sorry I couldn't help but laugh about a minute ago because I saw Libby had a comment. My favorite ROI, it's not me, it's them. I feel like that's the drop the mic. We've got nothing else to say. One thing we did not get to, maybe we can in the last minute. I think it's a subset of the broader story, but great practices for NDAs. I think probably it's safe to say, and keep me honest here, NDAs are a little more standardized for organizations, but they can hold things up a long time if they're not handled right, and sometimes they're just very high volumes so that you keep things moving. What have you learned from using the CLM, from using Evisort and practices for NDAs? How has that affected what you do?

Brittany Leonard (00:59:51):

If there's any provision that, I don't see it much in standard NDAs for M&As, I'll see one off provisions that have to be in there whatnot. But for standard provisions, I think the best thing about a CLM is seeing most of the provisions are pretty equal. And if one's an outlier being like, okay, why is this in here and is it really needed? And being able to say, Hey, here's an NDA that multiple people have been able to sign and making sure, please make sure just as NDA overall documents, please make sure your documents are mutual. I can't stand when I get an NDA that's like completely one-sided. Like we protect all our confidential information, but none of yours. And I'm like, why do I even want to do business with you if we're at that point at the initial stage? So I think that's a huge one to not only speed up the time to start your business relationship, but speed up the time for legal or anybody have to review it. Just making it mutual is huge.

Hal Marcus (01:00:49):

Great. Thanks. Well, we're just past the top of the hour. Brittany, Caitlin, Colby, thank you so much. This was a lot of fun. Hope everyone out there enjoyed and I will hand it back off to Lauren.

Lauren Magee (01:01:06):

All right, well thank you guys so much for a wonderful conversation. As I'm sure you can see in the chat and in the question box, we've had so much engagement today. This has been a great topic. So thank you all for your time and thank you so much to Evisort for sponsoring this event. And I know we've gotten some questions. This was recorded and everyone who was registered will receive a link to the recording within 24 hours. So you'll be able to go back, watch again and all of that. So be on the lookout for your email. Please fill out the survey when it comes up.

(01:01:44):

Right now. The CLOC's next big event is the Global Institute in May in Las Vegas. So we hope that you all will be coming to join us then. We're really looking forward to it. And then we will be back virtually in June. So we're looking forward to seeing you all again. Hope you have a great day.

Brittany Leonard (01:02:02):

Bye everyone.

‍