Hey, I see many folks trickling in. We will get going in the next few minutes as we hit the top of the hour. As we're waiting, Garrett, I'm curious, I know a lot of folks joining today probably remote and Netflix is a very innovative technology first company. I'm wondering, in the transition to remote, were there any fun things you guys were able to do? I'm thinking almost of being able to watch Netflix together with coworkers while remote or something, if there's anything like that you guys were able to do.

Garett Monroe (00:03:52):

With the transition was pretty seamless actually because I think being a tech first company, we were already kind of set up with laptops and phones and people worked from wherever. But you had to get creative just to keep it fun.

(00:04:08):

One of the things that we started, at least amongst the wider team that Legal Ops is under is we did our own version of MTV Cribs and that was our way of bringing everybody into everyone's home. This is where we're going to spend some time for the foreseeable future. We kicked off a whole MTV Cribs with the entire department and everybody shot little two to three minute videos and crib style right down to the fridge and the garage. It was pretty fun.

Memme Onwudiwe (00:04:41):

That sounds amazing. We might have to snag that. I asked that question with no expectation of such an awesome answer, but leave it to Netflix to of course. That's excellent. Awesome. I've seen a lot of folks jumping in. We're super excited to get started on our discussion. We'll just give it about a few more minutes just to allow some more folks to trickle in.

(00:05:09):

I know we had several hundred sign up for today and we're super excited to dive into this discussion. Garrett, just as I'm sure other folks are a little interested in that last idea you brought up about how you guys did that kind of MTV Cribs as part of the move to remote, I'm curious, was that one big event you did where everyone did that? Or was it kind of over the course of weeks each week, someone kind of showing their own home?

Garett Monroe (00:05:49):

I kicked it off and then you just nominate another person on the team. It was kind of like a every couple day cadence. I think it lasted, I don't know, the better part of a month before we got through everybody and it kind of traveled all around the world, all the way to our APAC team and it was pretty fun. I actually think maybe we should do a version 2.0 for this.

Memme Onwudiwe (00:06:11):

Yeah, especially with all the people that moved during the pandemic probably.

Garett Monroe (00:06:16):

Although having worked with you now for the better part of the year, I feel like you need your own travel channel. Every time I talk to you I feel like you're in a different destination.

Memme Onwudiwe (00:06:28):

Fair enough. Fair enough. That's likely a topic for a different day. But that is appreciated. Awesome. And also for folks hopping in, we've got a good amount of people. We'll give a couple more minutes for the last few folks to jump in.

(00:06:45):

I think one thing to also note is there is a chat and a Q&A available. We will be doing questions kind of near the end of the session, but if you want to centralize your questions in that Q&A field, that'd be helpful for us. And if there are some things that dovetail as we're going through the conversation, we'll definitely be getting those questions in too. So don't be shy.

(00:07:10):

Awesome. Hey everyone, we'll give it about two more minutes and then dive into kicking this off. We've got a good amount of folks on now, but we've got a lot of topics to dive into and I'm sure things that you folks want to discuss as well. And so we're itching to get going here. Excellent. Well, it looks like we come into a couple minutes past the scheduled kickoff time.

(00:07:57):

I am getting a couple green lights below going above that we should get going. Let's kick off with some introductions. Hello everybody. My name's Memme Onwudiwe. I lead legal and business intelligence here at Evisort and we're super excited to sit down and chat with Garrett Monroe today.

Garett Monroe (00:08:27):

I'm Garrett Monroe, I'm on the legal ops team. I've been at Netflix for just shy of eight years and the role I play on legal ops is leading kind of what we call the ops function. The liaison between legal ops and our legal team and then work with our op function to do these implementations.

Memme Onwudiwe (00:08:51):

Excellent, excellent. We're going to dive into the landscape of legal ops at Netflix and the changes that you and your team have been really successful in driving and kind of lessons that folks can take from that too.

(00:09:06):

Before diving in though, this is an interactive session, we would actually want to launch a poll really quickly just to get the lay of the land of who he got joining us today. And so everyone on the line, you should see a poll question hopping up just asking about what your current role is and based off of the lay of the land here, they'll help us align on the best place to take today's conversation too.

(00:09:35):

Excellent. We've got a lot of responses coming in now. It's looking like a lot of legal operations folks as well as lawyers and attorneys. And then a good amount of contract managers as well. And then about an equal but lesser amount of paralegals and the general counsel on the line too. So definitely a lot of folks who touch contracts and touch the legal operations space who are on today, which is definitely excellent.

(00:10:11):

I guess what would be great to hear Garrett would just be a little bit of what you're doing at Netflix, but then also when you first got there, if you could lay the lounge of how things were done from a legal ops perspective, how you guys were keeping track of contracts, et cetera.

Garett Monroe (00:10:30):

Yeah. I got here and we had what feels like a long time ago, we had three originals that felt hairy like, "How are we going to do this? This feels like it's so big." And as with any kind of starting point, there's a lot of infrastructure you've got to kind of build out and how are you going to work with the business.

(00:10:54):

So in working with the head of marketing legal at the time, kind of built out that what I call the 1.0 version, the launch. And then as the business grew and scaled and we continued to add more folks, it got to a point where we started to have to ask different questions.

(00:11:16):

It was no longer how are we going to do it, it was how are we going to keep doing it? There was an element of a tipping point where we had to figure out how does this look at scale? And once you start forecasting out if the current model works, we used to kind of pressure test it. We pride ourselves on being innovative. We're a pretty scrappy company despite our success and we don't rest. We like to test and push and test and push. We built it and now it's time to evolve it.

Memme Onwudiwe (00:12:00):

That makes sense. That makes sense. I know that you guys are working on a huge kind of planet scale. And so when you're talking about being able to scale this out and things being manual and from that kind of previous perspective, what would take you guys... A lot of the time, what were some of those bottlenecks that you guys are looking to scale past?

Garett Monroe (00:12:25):

Yeah, I think it's not unique to us at Netflix. It's probably every legal department, the contract is the source of information that the business runs off of. For us it just got to a point where the more and more originals we added or the more and more contracts we added, the more information you'd add to pull out of it and give it to someone else.

(00:12:48):

And so this was a regular function within the marketing legal team to pull information out and provide it to the business. It just wasn't scalable as a people and paper or even digital paper process. We had to really look at it and come up with a new way of how to get to scale and meet the needs of the business.

Memme Onwudiwe (00:13:12):

Now, that makes sense. That makes sense. I do want to dig a little bit into this kind of tipping point as you say because I'm sure a lot of what you said is likely familiar with a lot of folks on the call. And so far as we have to manage thousands, tens of thousands, hundreds of thousands of contracts. People could ask us any question about any of them at any point in time and we may be track kind of five things manually within there. This is a headache.

(00:13:38):

But I think going from that pain to actually saying, "Hey, we're going to be concerted and do something about it." Is an important threshold to pass. Before diving in there, I did want to ask the crew and the group that we have joining us today where they are on that legal technology journey. We'll just have a quick poll question coming up just asking whether or not your company or firm is leveraging a legal technology today.

(00:14:15):

Excellent, excellent. We're having a lot of responses coming in. Looks to be around 60/40 with the 60 using some sort of legal technology and of course that was vague on purpose. We didn't really want to box people in too much and with about a 40% moving forward without that kind of technology.

(00:14:40):

And so understanding that, this is probably for that 40%, that's probably a lot of people who are having in that similar phase that you were talking about beforehand, kind of drinking from a fire hose, was interested in learning not just about what was that tipping point, but how did you work to get internal alignment on the need to even get a tool?

Garett Monroe (00:15:04):

Yeah. We put together a task force. It was like, "Hey, how are we going to keep doing this at scale? What does this look like in the future? How can we improve it?" And so we had the people that have been involved in this, it's their tool, their job duties and put them in a room and it was like, "Let's throw ideas at the wall." What I found was folks were coming up with little incremental wins, little small tweaks to improve it, which I think collectively would've made some improvement. But it wasn't big sweeping scale changes.

(00:15:46):

After a couple of sessions of not really seeing that ideation, that creativity be unlocked in a little bit of a provocative move, I tried to move them a little farther from the narrow thinking to... At the time I just threw it out there as an idea, not knowing it was even possible, but basically look like I want us to be so creative that... Why can't we just have a machine, do this control, find experience that you guys are doing and extract the information for us?

(00:16:19):

And everybody kind of laughed it off like, "That's not even possible. That's not a thing." And so I kept actually saying it to everybody that I met and when Jenn McCarron started, I had sat with her and Jim McCarron leads the legal ops team at Netflix and I sat with her and I actually pitched this idea to her. In true Jenn fashion, she looks at me and she goes, "I've got an idea."

(00:16:42):

I was like, "See, I knew it was possible." That kind of led us on this journey. That was really the tipping point. It was let's go out and explore and find out what's possible and see if we can actually meet what your crazy idea is. That led us into the world of AI machine learning, the natural language processing platforms and we kind of canvased the landscape and we just looked at who was and what were the opportunities and what they could do. And then we just, what Justin at KPL calls the SmackDown, we just put them all together in a wrestling match and tested them head to head and just looked at the outcome, looked at the partners and tried to figure out who's going to get us to the place we're trying to go. It was truly kind of just a test.

Memme Onwudiwe (00:17:34):

Yeah, yeah. No, that's huge. I do want to focus on that because I think that we're in a world now where legal ops teams and lawyers need to evaluate AI and that's not something that they teach you as you're kind of coming up into those roles. And so the thoughtful way that Netflix approached evaluating AI systems and actually taking it internally and testing it live, I think that's something I do want to dig in on.

(00:18:03):

But for the sake of also keeping it interactive with the teams, I do have one more question for folks, which is have you had to make a business case for buying new tools or technology for your teams. Which is what we kind of just talked about is that experience of hey, I know we are feeling the pain because we have way more contracts than people and we expect it to know everything in there but legal... It might be a cost center of the company. You need to talk through, "Hey, why do we need this tool?"

(00:18:38):

And so whether or not you've had to formally do that, I think it'd be interesting how it looks like. Thus far about 80%, looks like about 80/20 do have to create these kinds of business cases for buying new tools or technology for teams. It might be worth us talking a little bit about what that process was like to Garrett. But we can start with just the process of evaluation that you undertook internally and what that SmackDown really looks like for any folks out there who are looking to set up SmackDowns of their own.

Garett Monroe (00:19:15):

We were fortunate enough that this scenario that we were trying to solve for was one that I helped build originally. It was a very established process and we had a lot of consistency in the process and the people that were involved. One of the things that we focused on is the people process tech. If you do that kind of order, it generally doesn't work. If you go tech first, you don't really have a process to wrap tech around.

(00:19:45):

Typically, you have to have the right people and a good established process and then you can augment it with some technology. Again, not a total 100% formula but a good kind of guiding light. So since the process was really well established, we knew what the input was, we knew what the output was and we knew what we were trying to attain.

(00:20:07):

Basically we just took a piece of that and tested it with each platform and then brought in the stakeholders to evaluate the output. We kind of went through that whole test from what would this look like, can the tool perform? Can it recognize these clauses? Can it bring it back? What's the level of consistency and accuracy? Can we trust it?

(00:20:32):

What do the teams look like that we're working with? And then one of the, I think key components to this whole thing was we were new to this as well. I didn't know anything about AI before I started this journey. It was my wacky question and then it led me down this path. And then partnering with KPL was, I feel, like a really big accelerator. They were new into it as well, but they were working with other partners and working with Evisort and that helped really allow us to accelerate our learning from their learning.

(00:21:08):

I think anything that you're bringing in from a technology standpoint, you're not entirely positive it's going to work. There's something about your unique case or the way that your business runs or your internal systems, all of those things play a part into the success of the tool that you're evaluating and you have to be willing to fail.

(00:21:30):

Netflix is pretty big on testing, gives us the opportunity to have crazy ideas and go out and try. We knew we wanted to try something better and we were willing to test it and learn into it and if it didn't work, it didn't work but at least we explored it. So you have to be willing to allow something not to work in order for you to find a way to make it work.

Memme Onwudiwe (00:21:52):

No, I really like that. We thought your approach was thoughtful at the time and especially for a tool like this where it's inherently about ingesting documents or tools never seen before and having the ability to extract out information that's relevant to push the downstream systems. I've seen some folks who would undergo that kind of purchase without actually testing to see hey, can I send you something? And it does that in two seconds.

(00:22:25):

I think it's really key that if that is what you are looking to buy and purchase and the goals you're looking for to actually see it happen live in the Missouri, the show me state. I know you guys aren't based in Missouri but Netflix might as well be because you guys are really pushed on the show me and I think that's a great lesson for all.

(00:22:43):

Excellent. I do want to dive into, because I think a lot of folks, they get to the point of procuring a tool but then when you think about hey how do we actually drive impact internally and how do we actually make sure that this is being used in all the places it should be, especially in large organizations that can break down. I do want to talk about that kind of post bringing on a tool how to onboard and evangelize and use it correctly.

(00:23:11):

But before that, did just want to touch base on with the crowd on how many of them have had experience in evaluating legal technologies that leverage artificial intelligence. Because I know we just talked a little bit about some best practices and doing a SmackDown or something like that and wondering if that's something that lots of folks here have experience with or if that's something that it might be a bit more novel to the participants.

(00:23:44):

Interesting. Yeah. It looks like we're about, you could call it 60/40 or 50/50 as these numbers keep bouncing about. Seems to be settling more towards the 60/40 with about 60% of folks saying that they do feel comfortable evaluating legal technologies that leverage AI and about 40%. We have close to half saying actually they don't feel comfortable doing that.

(00:24:13):

So hopefully for that 40% this has brought some clarity and once again I do see some questions coming in, but if you do leverage the Q&A section, you can ask some additional questions as well and we'll be weaving those in and addressing at the end of the call.

(00:24:29):

Excellent. But to pass it back to you, Garrett, I think that would be really interesting to talk about what would be after you found the tool you wanted, how did you get that buy-in internally, how were you able to get this to the different teams that could leverage it? And then also I see that some vendors here too. Was there anything that we did to help or things that you think a vendor could do to help as folks do this? Because SaaS is software as a service.

Garett Monroe (00:25:04):

Once we landed on utilizing Evisort for our after the test and we went into basically a pilot where we took it to the next level of trying to prove out that Evisort could help do this extract that we were looking to do, it was about a four month POC that we partnered again with KPL and basically you learned together.

(00:25:29):

We went through and trained these models. Again, we were learning as we were doing this and we were having some really good success and then we got a little arrogant and we're like, "This is going to work." And then we really got specific and then we kind of in essence broke it. We actually started getting a decline in our results and then it was another learning. This whole time, it was kind of true to the Netflix culture as a feedback session.

(00:25:57):

It was like every time we got in and tested, we learned something new. As we got better and better at training the models and working with you guys, Amin was critical in this and you were critical and kind of helped guiding us in some best practices of how the algorithm actually worked and how to train the models properly.

(00:26:15):

We got to a place where we started getting consistency and it kind of proved out and as we started throwing more at the tool, the better the accuracy and the more consistency we were getting. We realized that when you compare it to what the human was doing and what Evisort could do, even with this testing model we were doing because we're not fully connected, so we were hand inputting these agreements in, we were seeing time savings in the 30-40% range and each time we went through this contract clause extraction. We knew we were onto something. It was kind of this idea that one, we were saving time but we took a static document and turned it into data.

(00:27:00):

We're no longer in this PDF that's control, find and OCR. We actually now are getting data back out of the contract in the form of clauses that we care about. So then it starts unlocking the thinking of potential. We look at it and said okay, "Great. We can do this. What else could we do with it?" And so it starts just open your creativity. Could we push it to other systems? Can we just replace our output, which is a current... It's a Google Doc, what else could we do with it?

(00:27:29):

When we got through the POC, the net outcome we is what we knew something was good here. And so I credit Jenn McCarron with this. If anybody's ever been around her in any of the Clock events or anything like that, she's the ultimate presenter. And so she pushed and said, "You've got to get out and present this and you got to tell legal about this tool."

(00:27:55):

And so I think we spent about six months road showing the Evisort experience and results and the benefit of that in Jen's wisdom was right. It unlocked all kinds of inspiration for people to think about where could we utilize this, where else is this beneficial? What other areas the business could we test into. So I think it unearthed somewhere in the 15-16 ideas right away. We just started exploring what other potential is here and how can we leverage Evisort and the model building to speed up the business. But also again with the goal of this is turning contracts into data.

Memme Onwudiwe (00:28:43):

I do want to dive a bit deeper into the turning contracts and the data because I think that's something where we aligned a lot on vision and I think you once said turning contracts into data is integral because it helps you take that data down API information, super highways is I believe what you said and I still try to use that every time I can from the moment I heard you say that because that's exactly the kind of thinking that we like to see from a legal operations teams.

(00:29:12):

But I guess before diving in there, I think what was really impactful and helpful about what you just said was the fact that you were seeing 30 to 40% in time savings, as you guys were testing. And that's important not just because those are great numbers and they are, but also because it's important to look at the impact of technologies that you bring in as you're implementing them.

(00:29:35):

I guess a question that we're about to ask and put on the screen for folks listening, are do your teams go and look back to measure the impact of legal technology solutions that you've implemented in the past? I'm curious about that. Maybe we can talk about what that even looks like. But especially when it comes to tools like this and especially as we said before, legal being a cost center at a lot of companies and you have to build that business case wondering how much that means on the back end.

(00:30:09):

You have to almost prove that business case that you built out in the first place. Okay. It's looking like another 60/40. Yeah, we've got about 60% of folks saying, "Hey, we do look back and measure the impact of legal technology solutions." And about 40% of people saying that, "No, that is not the case." Okay, interesting.

(00:30:40):

Excellent. Well, I guess I think what would be interesting would be to dig in a little bit deeper on that idea of turning contracts into data. I think it almost goes back to that initial question you asked of hey people are looking at how to control F faster, but why are we even controlling F for that information? Why is that even the paradigm through which we're collect data from my contracts and almost shifting how you're even thinking and approaching the issue from the start.

(00:31:07):

We see a lot of companies... When we start turning their contracts into data, kind of like you, they almost don't know where to start. So it's not a capacity that they really had before. I would love to learn a little bit of almost your guys' experience in taking those tools and as you're road showing it what different teams are saying and how that's almost shaping how you view the tool in its capacities.

Garett Monroe (00:31:32):

While I haven't done this, I've always wanted to do it which is basically you take a contract and find out how much time is spent throughout the business taking the contract apart. Finance needs it, insurance needs it, all your different department needs it. They all need one two clauses, what have you, different data points to do their job. So the contract again is the product and all these folks are taking pieces of it and I think we would be pretty surprised at the amount of time spent unwinding a contract after it's actually signed.

(00:32:10):

I look at it and I see the potential of we have all this stuff locked in an agreement, all these people that need these data points, this test kind of proved out that there's the ability to turn it into data and push information to the people who need it as opposed to them having to go find the contract, go find the clause they care about, the contracts, 50 pages, they care about two paragraphs.

(00:32:34):

It just really unlocked the thinking of a new way to get people their information. It really... The test unlocked creativity. I mentioned to you before and people that know me, I use this quote all the time is Henry Ford had said, "If I asked people what they wanted, they'd just say a faster horse." And then he built them a car. And so there's times where folks get really consumed in the way that they do it and sometimes it's hard to see outside of the way they do it.

(00:33:05):

I think teams like the legal ops function get the benefit of sitting back and seeing it from a different perspective. And so working with you guys and KPL really kind of is like this brain trust where we just throw crazy ideas at the wall and see if they stick. The data concept I think this is where we're going. We used to watch movies on DVD and now we're streaming movies. So it's eventually going to be a non-static document. And so for us, I just look at it as this is the step that we needed in order to take that contract to a different level and unlock all that information and push it around.

Memme Onwudiwe (00:33:46):

No, that is definitely key. I am seeing a lot of questions coming through. We can't take a little bit of time to answer some, I think someone asked and they're anonymous, "Who in what is KPL?" Which is probably worth... Just because you have brought them up a couple of times.

Garett Monroe (00:34:05):

So Justin Hectus leads KPL and KPL is the tech arm of a law firm. Pretty unique set up but KPL offers basically managed services. They can do quite a suite of things ranging from workflow automation and other things. I use Justin and team for all kinds of things including sometimes therapy.

(00:34:30):

So they're open for all kinds of things. That being said, I think Justin might even be on the call, but if you haven't met him, definitely reach out to him. He's a great bridge in the legal ops community. He works with a ton of folks outside of Netflix obviously and they're a thought partner that really helps us augment. If you're a team that's pretty lean, a team of one, a team of two or even if you're a team of 20, they just help augment what you're able to do by some outsourcing. It's a great partner.

Memme Onwudiwe (00:35:13):

Awesome. Excellent. Yeah. A Big fan of Justin, KPL and team as well. Another anonymous attendee actually is asking, "What type of contracts are you using this AI for?"

Garett Monroe (00:35:26):

Yeah. We've ran a host of contracts through it. It's been talent agreements and production agreements, real estate agreements. There hasn't been anything we haven't thrown at it and we're I think maybe half a dozen kind of what called POCs in. But the main ones right now we have tested is the other ones that I mentioned.

Memme Onwudiwe (00:35:51):

That makes a lot of sense. We actually have two questions from Daniel O'Brien. The first one being, "Did you experience any resistance to the changes and if so, how did you deal with that?"

Garett Monroe (00:36:05):

There's always pushback. I think the common pushback that I think I get is, and it's usually said in jest, but I know there it's rooted in a little bit of concern is, "Are you going to automate us out of a job?" And I think the answer to that is no.

(00:36:21):

I always tell legal professionals and lawyers, my response to that is, I'm actually trying to free you up to stop doing things you don't even want to do anyways. Nobody really wants to go control+find. That's just a crappy experience. So the idea is I'm trying to free you up of things that you don't want to do to give you more time to do things that you're actually really good at. By way of example, lawyers didn't go to contract search school. That's not a thing. They went to law school to be lawyers. I want you guys to be in that space doing more of that. I'm just going to help you get the information quicker.

Memme Onwudiwe (00:37:00):

No, that makes a lot of sense. And then actually Daniel's second question, I can actually help out with it was around the idea of using outside groups to process your contracts instead of Netflix itself kind of leading to potential data privacy risks. Yes Daniel, that's definitely a concern folks have seen, especially if you have contracts that say they're confidential and then you're sending it out to maybe a team of people in India per se where folks are manually going through them.

(00:37:35):

There have been some concerns there, which is why folks do like using systems like AI where you don't have to deal with issues of third party people touching contracts that seem to be confidential. Excellent. And then I guess we have a few more questions coming in. I do want to ask you one thing before diving back into the audience but audience do definitely keep them coming. These are excellent questions that we'll get to.

(00:38:00):

I wanted to ask, actually to the audience first, does your company have the ability today to turn your contracts into data? Obviously, you can have a paralegal go into a spreadsheet and type all your expiration dates. But if you have a question about your thousands or tens of thousands of agreements, let's say you track five things in them, if all of a sudden you need to track a sixth thing, is that an internal fire drill with all your teams reading every contract and control [inaudible 00:38:31] and scanning?

(00:38:32):

Is it sending it out to an outside council so them or some team can send them off to India or some other place to be reviewed by hand. Or is that something where your team internally has a tool that can quickly allow them to get to that extra data point in your contracts even if it's not something that you have been tracking historically like a force measure before 2020 or something like that?

(00:38:57):

Excellent. It looks like this is one where it's still a gap. Like everything else today, it's about 60/40 but this time only 40% of people saying they have the ability to turn their contracts into data and with a 60% of folks saying maybe they do not have that capability today.

(00:39:20):

That's definitely helpful. There's a question here from Michael about and a little bit on that point of turning contracts into data, there's a question here about whether or not you can extract certain types of... Whether or not you can train the AI to find things you're looking for. I know that was part of your testing, not just out of the box but also the AI and training for new things. If you could talk a little bit about that experience and how that was something that you actually tested for before even purchasing a tool.

Garett Monroe (00:39:59):

Memme would you do me a favor and repeat the question because I was reading the questions that were coming in. So sorry. I was multi-tasking and I wasn't listening but I was reading.

Memme Onwudiwe (00:40:09):

No worries, no worries because I was reading as I was reading that and we have a lot of questions coming in, so I definitely understand that. But it's basically a high level question. I was trying to merge a couple of these together because a few that talk about the same thing but it's basically a high level question about hey, out of the box algorithms are great, but how is the experience in training a new algorithm? I was going to ask you because I know that you guys actually tested that extensively before you even purchasing a tool, how you tested that capability as well.

Garett Monroe (00:40:40):

Yeah. It's a great question because we went into this not really knowing a lot and so we didn't know what to expect or how fast it should be. In the head-to-head, in the SmackDown that I called it, we basically worked with each respective platform to kind of test and train on the language that we were looking for and then we would feed it new contracts to see if it could pick up on it.

(00:41:10):

It was nothing super scientific. It was kind of our best guess on how we think this would work. As we were kind of doing this incrementally, Evisort hit pretty quickly and it was a pretty quick response to the language that we had trained it on, that it picked it up on net new agreements that we put in, which we thought was a pretty good indicator that the algorithm was pretty powerful.

(00:41:36):

Out of the box, Evisort provides, I don't know, somewhere around the 30-ish out of the box provision that you guys have already trained on. But force majeure and confidentiality and things like that weren't provisions that we were looking for. We were actually looking for very specific clauses that teams needed and so we needed to train on that. It wasn't something that Evisort had trained on and this is where working with Justin and KPL was really helpful was kind of going through this feedback loop and training.

(00:42:07):

So that custom model building, it took a little while to figure out and understand and working with Amin on your team to better understand how to do it. And again, I had mentioned that then we did it wrong and we broke it and we had to go back in and unwind some tags. But ultimately, that custom algorithm building, it's actually not as complex once you understand it and it sounds super scary. You're like, "Oh my god. I'm going to train algorithm and how does this even work?"

(00:42:34):

But the reality is kind of as simple as highlighting. You just really need to have an informed captain, a person who knows what they're looking for. What I've found that's helpful is usually these groups have some sort of output. So understanding what their output is and then using their inputs to help drive the output was I think a critical moment for us to understand if you can train with the output and then bring inputs in, that seemed to be kind of an accelerator for us as we learned.

Memme Onwudiwe (00:43:07):

No, that's huge. And then I think the collaboration and then also going in and being able to see pretty quickly, hey this tool as we're training for these use cases seems to be sticking so we can be confident that as we take into this new green field, it'll be able to tackle that too is a great evidence based approach.

(00:43:31):

Also, here's a question here from Amy. It's asking, "Did you have a robust CLM platform prior and what were the pros and cons of updating?" So yeah, it might be helpful to give them kind of the lay of the land of what you guys had beforehand.

Garett Monroe (00:43:45):

Yeah, so while we're a tech company at heart, our tech is on making sure that all you people enjoy what we're providing on the service and less about the technology that drives contracts. We have a homegrown CMS system that we have and then outside of that it was lots of what you expect to see is Word and Google Docs and not a lot of workflows.

(00:44:11):

So we do now have various things in place, but this platform was really the first thing that we were bringing in that would ultimately be a pivotal kind of change. Again, I mentioned that it unlocks the thinking a little bit. In this again provocative moment, I had just said what... Hey, I even remember on the call I had asked Memme, "What else aren't we using? What aren't we using on your platform that we should be using?"

(00:44:40):

I felt like we were pretty isolated in this clause extraction experience. I think that was the "flip the switch" conversation of what if we just turn it on? What do we get and how do we benefit?

(00:44:51):

It just really forces us to think really big. It was like what else could we leverage? What were some of the pros that we could get and what if we use it as a replacement for our contract management system and what if we use their workflow pre signature side of things?

(00:45:06):

And so again, it was just you have to get pretty creative in how you think about utilizing a tool. Not everything's going to fit everybody. I think it's hard to have a one size fits all platform but you use it for what you need and then you find out where else you can expand it. Sometimes the only way I think you can get really creative in that thought is to ask a really provocative question and then realize, oops, maybe I went a little far on that question and then you kind of bring it back. Or maybe there's actually something there that you can start to look into.

Memme Onwudiwe (00:45:40):

Yeah. Wow, that was great. What I really liked about that too was that idea of asking that big question that hasn't been asked before. Because you just even spark something in my mind. In reality what we're almost doing here is drastically reducing the incremental cost of tracking one more data point in your contracts.

(00:46:01):

If you've been thinking about it as, "Hey, we track these 10 things in our contracts." You probably came to those 10 not because of the only 10 things you need. That's because you're weighing the cost of asking someone to track that in every single document versus the benefit of having it on hand.

(00:46:16):

But if you can get 60 data points or a 100 data points across all your contracts with kind of no lift, then maybe you do track force majeure just in case a pandemic's on the horizon or maybe you do track every increase in costs, clause even for contracts under $50,000 if you had a policy beforehand of over but you only had that policy because it was too hard to do.

(00:46:40):

If people take anything away from that, it's that dreaming bigger and actually thinking of a different paradigm and those questions that you didn't know you could even ask and then beginning to ask them, that's that. I just love that perspective.

(00:46:54):

I guess there are a lot of folks questions coming in. We'll do one more and I do have a couple questions for you myself, but I do want to respect the folks. This one is actually a great one. From Anonymous, was AI your primary focus and then did you have an automated workflows in place? Did you use Evisort for that? I know that you guys started using us for our AI extraction and analytics and contract management, but did actually last year begin using our workflow tool as well? And so talking about a little bit of how you guys are using that I think could be helpful for this question asker.

Garett Monroe (00:47:31):

Yeah, so while we love working with Evisort, Evisort's not the only person we work with. We look at it as what is the solution. I'm not trying to fit every platform into every problem and really what we're trying to look for is what opportunities are out there and what is in our kind of suite of tools that we can deploy to help and makes that the best scenario for that particular opportunity.

(00:47:58):

On the workflow side, we're testing with a couple of pretty repeatable workflows where the business is capturing data points and entering into a form. By way of example, one of the low hanging fruit one would be an NDA, these are pretty repeatable, pretty high volume depending on where you're at in your policies.

(00:48:21):

It could be highly negotiated or it could be not negotiated at all. I think the workflow is again really dependent on an established process. You need a really clear start and end point and something that's highly repeatable.

(00:48:38):

If it's a very one-off and highly negotiated, maybe the workflow's not the way to go. But I think our team's really good at intake and looking at opportunities and evaluating what is the best fit and rather that be an Evisort and their suite of offerings or is there another solution. Look, unfortunately not everything is also techable. Some things are just better off in a manual state or temporarily in a manual state until something better comes along.

Memme Onwudiwe (00:49:13):

That's excellent. There's so many questions. We're just going to run through some of these. You guys win, you guys win. This one actually feeds into a little of the one you just answered because you were really talking about Evisort almost in an ecosystem of tools, which is a good way to think about it.

(00:49:30):

I know we've been very forward on APIs for that. And this person's asking, Hey, how did you integrate this with the other tools and databases in your company? And you can talk a little about maybe some of those APIs or just other ways that the way you said kind of plays with the rest of Netflix's toys.

Garett Monroe (00:49:46):

API integration is going to be critical in order to connect this kind of Frankenstein system where if you're going out and getting kind of... Think of it as if you're assembling a team, I'll use baseball. You're getting the best third baseman and the best second baseman and the best pitcher and you have to assemble the team.

(00:50:08):

And so if you're looking for very, very specific types of platforms to fill a very, very specific use case, those things are going to eventually have to connect. Otherwise, you're locking everything in a room. We're actually shoulder deep into connecting some internal systems with Evisort right now, which then the benefit of that is now we have a free flow. It's all time anytime query of Evisort for our contracts as opposed to, as I mentioned earlier, kind of taxiing them over for these POCs.

(00:50:43):

While we've been in the POC environment for a handful of these opportunities, this unlocks the movement into the more automated space. So now we can bring business users in, they can actually work within Evisort to get the information that they're looking for because it's always real time. And then again, remember I mentioned it starts unlocking the thinking and then the next part of that is how do you bring data back into tools? So back to that push.

(00:51:15):

An API is going to be critical. I think thus far what we've experienced with Evisort's API is that it's pretty rich and nimble and we haven't had many roadblocks in connecting them.

Memme Onwudiwe (00:51:29):

Yeah. No, I think that that's a great answer. Frankly what's really key to leveraging any API and what's really key to leveraging any contract tool in an ecosystem is turning the contracts in the data. Because you're not going to push that scanned PDF to Salesforce.

(00:51:45):

You might, but what you want in Salesforce is the expiration date, is the payment terms, is the actual information that's kind of taken from that document. That's excellent. This is actually a great question coming in from Jordan Cavitt. It's a question about who internally at Netflix were the most enthusiastic adopters of Evisort? So we talked about this road show the different teams, but what teams were the ones who were raising their hands and excited to get going from day one?

Garett Monroe (00:52:22):

If anybody knows Malcolm Gladwell, you have early adopters and those are usually the people that are jumping up and down and they're kind of spread throughout the company. Right now I think Netflix has quite a few early adopters just as the general type of people that we hire.

(00:52:38):

But that's not also the case. When you talk about people who have been in the entertainment business for 15, 20 years, they're accustomed to standards and practices that come with that business. But I've actually been pretty impressed with the level of excitement around it.

(00:52:59):

I think anytime you mention something can save someone time and also take away the remedial mundane type task of a control+find or tracking, I think everybody goes through this tracking mechanism where they're tracking different data points and you have spreadsheets everywhere.

(00:53:19):

This really unlocks the ability to maybe minimize some of that tracking again back to the data point and then minimize just some of that more mundane work. Spending time while we get good at it. I think any lawyer and legal professional on this is pretty dang good at the control+find and keyword searching. This just really speeds it up and I always compare it to a car.

(00:53:46):

It's like, I still need lawyers, I still need legal professionals, I still need everybody in the business. I'm just going to put them in a car that goes really quickly.

Memme Onwudiwe (00:53:56):

I like that. I like that. I also like that idea of people do get really good at that control, laughing and things like that. But I remember when were starting Evisort, what we always joked was why do we even call the contract management system if you have to retype all the information to manage it? You're the one managing it. The system isn't. It's just kind of reminding things that you fed it. So I love that idea of thinking fast.

(00:54:22):

Well, another question here from Jordan actually, which is actually is just looking for an example of a custom algorithm to build. What kind of clauses... Or maybe even if you could dive in and not too deep of course, but just to potential use case of using a customized Evisort algorithm that's kind of helping with Netflix. SoI mean without going into the secret sauce, in and out doesn't disclose their sauce, but I'll just give you an example of every contract has some element of payment terms. While there's a contract value payment terms are important for finance and finance has to go in, they've got to figure out the payment terms within the schedule that's either tracked or they do it manually and it's a control+find, they go to their payment schedule.

(00:55:15):

What I think for a use case scenario would be is training the algorithm to go find the payment schedule to go find the payment term, to go find all that information that the finance team is going to need. And having that just be in a repeatable process. So as the contract goes through, it's already extracting that data point and now the finance team actually just goes to get the data point as opposed to control+find.

(00:55:39):

And look, everyone knows lawyers don't always draft in a very linear fashion, sometimes purposely. The form might be our form, which is great. That means I know exactly where it's going to be. But what if it's third party paper? Now it's in a different location. What if two different lawyers drafted it? It's in a different location or worded differently.

(00:55:56):

So the control+find while is good, it's not always foolproof. Whereas I think when you train the algorithm and you start getting those really high percentage results, you can actually get to the clause and the information way faster than you ever would have unless you were the person that drafted it. The person that knows it in their mind is obviously going to always be faster. But when you get the algorithm and you get volumes of agreements, I think that's where you really start unlocking that potential.

(00:56:26):

No, that's excellent, that's excellent. I guess to pivot a little bit and I know we're coming in the last 10 minutes here, did you want to talk a little bit about you in your career? I mean as the legal ops function is one that's kind of really evolved in the recent few decades. And so curious about your role and how engagement with AI.

(00:56:50):

I know that you're the AI guy there internally now, which maybe wasn't the case before. So for other legal ops folks kind of looking at their careers I think could be a interesting just to hear a little bit about that.

Garett Monroe (00:57:01):

Yeah. I know you call me that I'm not even sure I think of myself of that. I'm in school. I'm still trying to learn all the potential capabilities of this platform and AI in general, but definitely a lot more comfortable than I was when I first started the journey. Legal ops is what I guess as a true industry and actual titling is maybe what, 10, 10-ish years old. I didn't start my career in legal ops.

(00:57:35):

Somebody had joked that it was like everybody does legal ops until they actually do legal ops and then it's like, "Nah, I actually doing lawyer work or contract manager work." But we have all moonlighted over the course of our careers and being in legal. I think I'm no different. I started off as a paralegal and then into a contract manager role.

(00:57:56):

And then I think the opportunity here was building out the infrastructure of a department. I hadn't done that before. That really kind of started flexing a lot of my ops chops. And then fast forward to Jenn McCarron coming on and then establishing a team within Netflix. And there was just a lot of opportunity.

(00:58:19):

Again, an appetite is helpful. Netflix is a tech culture at heart and so we're always looking to innovate and find new ways to do things. And so having that push for innovation and creativity helped my career because it allowed me to test and fail and be creative and come up with wacky ideas. Like, "Let's have a machine do it." And then the AI piece of it has opened doors internally as far as it's helped me grow my career here, it's helped me help the business.

(00:58:52):

At the end of the day, it's really what's best for the company and for the business, it's less relevant what's best for me. But as a byproduct of that, I get to work with, I think Justin at KPL, I know I've mentioned him a few times, but again, he's been a big partner of this, but he had sent out a Christmas card and said, "Working with cool people to solve hard problems." And I was like, that is exactly what my day is like. It's a whole bunch of cool people and everybody's got something they're trying to solve. And then we're over here putting our heads together and aligning with folks like me and team to try to match a technology to an opportunity. It can't hurt. It definitely can't hurt to learn this and test into it.

Memme Onwudiwe (00:59:38):

Excellent. No, that's awesome. The way you sum that up from Justin's quote I think is exactly how we think about the collaboration here too. I know we are winding down on time and there are a lot of questions still in queue, including Garrett, "Are you hiring?" Which we can leave that person to follow up directly.

Garett Monroe (01:00:00):

The answer's yes, we are hiring. We're always looking for more people. So yes.

Memme Onwudiwe (01:00:06):

That's excellent. Well, I guess before we do head out though, as we saw from the initial poll, lots of legal ops folks here, lots of folks from in-house legal, lots of people probably where you were in the beginning of this call, which is describing that place where you just know you've got to do something, you've got to start looking for something because the way that things are going, isn't where it is.

(01:00:29):

Any kind of parting thoughts to folks who are in that stage about not only going out and evaluating a new tool, but also just kind of what their goals should be or just what their mindset should be?

Garett Monroe (01:00:45):

Be bold, courageous, and be willing to take some chances. It's not all going to work and I think everyone's got to get comfortable with that, including the person leading the change. If you have to bring this back to your GC or whoever you're reporting into and say, "I have this idea, it might not work."

(01:01:05):

You have to be willing to have that not work, but there's some lessons that come out of it. So we didn't know if this was going to work. In fact, we don't even know if it's going to work. It's all kind of evolving. We know things are working, but we haven't pieced it all together. And by no means are we at full mature state. The Evisort team continues to evolve their platform and new things hit the market all the time. Always just stay flexible, lean in and give it a shot and see how it works.

Memme Onwudiwe (01:01:39):

Awesome. Excellent. Well, yeah. Thank you so much everyone who joined today's session. We will make a recording open as well. Thank you so much, Garrett, for your time today talking about the experience of our work over the last several month together and also giving insights into the [inaudible 01:01:58] topic. I know its very hot right now, which is not just leveraging these technologies, but evaluating and onboarding them.

Garett Monroe (01:02:07):

Thank you for having me and thank you for all the folks that showed up and asked great questions. I still am little humbled here that I'm no AI guy, I'm just a person with a wacky idea.

Memme Onwudiwe (01:02:21):

Awesome. Excellent. Well, hey. It's driving 30 to 40% time savings from day one, so I think you're more than just that. But thanks so much everyone for joining today and have a great weekend.

Garett Monroe (01:02:38):

Thank you all.

‍