Rolling out Legal AI Assistants at Scale. A discussion with Tom Rice.

Tom Rice is the director of Legal at TravelPerk, one of Legal OS' first customers. He joined our co-founder and CPO, Jake Jones, in our first webinar of 2024 to discuss the opportunities and challenges of rolling out AI at scale.

An abridged transcript of the discussion follows.

Jake Jones: There’s no one better placed to discuss rolling out AI at scale than Tom Rice, TravelPerk’s Legal Director. We’ve been working together since last April, which was less than a month after the launch of the ChatGPT API. Since then, Tom’s vision for the product, for generative AI in Legal, and his lean product mindset have been massively influential in us developing a powerful and reliable product that has already supported over a hundred TravelPerk colleagues with tens of thousands of queries. Great to have you here, Tom. 

Tom Rice: Thanks, Jake. What a nice introduction.

You contacted us in December [...] and our first beta test showed a lot of potential. Then GPT-4 came, and we rolled out what has become our de facto corporate Wiki, available through a Slack channel.

Jake: My question to you, to frame this whole session, is what are the main use cases TravelPerk is currently automating with generative AI... and how did you get there? 

Tom: When we started conversing [in March 2023], we had a previous decision tree-powered bot designed to automate very low complexity work, trying to push people into workflows and release company information on demand. And so we were very interested when GPT-3.5 hit the market last November. You contacted us in December [...] and our first beta test showed a lot of potential. Then GPT-4 came, and we rolled out what has become our de facto corporate Wiki, available through a Slack channel. It’s called the #ask-legal bot, but it’s actually no longer just the legal team powering those responses with content; I collaborate with Privacy, Security, Revenue, Bids, and more teams to power those responses. 

When you’re thinking about how to deploy legal services at TravelPerk, you’re clearly prioritising the experience of your sales folk, marketing teams, and deal desk.

Jake: It’s interesting to think back to the early days between 3.5 and 4. We were working with a member of your team called Gabi, and he uploaded your publicly available privacy stuff. That was an early proof of concept. What really drew us into partnership with you was the perspective you had, which is putting the users you serve, the end users, first. When you’re thinking about how to deploy legal services at TravelPerk, you’re clearly prioritising the experience of your sales folk, marketing teams, and deal desk. Where did that mindset come from, and how has it informed your implementation of Legal OS? 

Tom: Let’s just say I’m passionate about all things product. One area I always try to provoke debates around is end user experience. Particularly in the operations world, you’re really there to obsess over end user experience, not about what the lawyer wants. The lawyer will provide the content and expertise, but it’s actually your job on the operational side to take that and create magic with it on the other end. We’ve always been preoccupied with the “launch, then learn” mindset, because you can stifle yourself trying to contemplate how people will interact with your product, but the reality is often quite far removed. 

Jake: What I found particularly interesting about going back to the end user was looking at how Legal functions within a whole organisation to empower that whole organisation. And having that perspective opened up whole new opportunities for us. It’s what allowed us to look at the relationship between Sales and Legal, and what lived experience of Legal the sales folk have. And it was clear that you and Andy shared that perspective. It was interesting what you then chose as your first use case. We see a high volume of entire RFP questionnaires getting pumped into the bot, sometimes 20 questions in a single message, and the bot handles it. Where did that come from? Was it an intentional decision, or is that just how people began using it? 

Tom: It’s a joyful unintended consequence. In a business like TravelPerk, at this time last year, completing a big enterprise RFP was multi-stakeholder and multi-touch. You would walk to the privacy team, to the security team. What this technology allows you to do is aggregate the upper level. We, as five different teams, don’t need to come in on each individual transaction. We collaborate on the content layer, and as a byproduct we collapse the stakeholder stack. The consequence of that has been unreal in terms of efficiencies. The time investment of all those stakeholders getting together on the individual transaction layer has evaporated. So the cost of potentially acquiring or failing to acquire a customer has been greatly reduced. That’s been a real win for us. 

Jake: And it was amazing for us, too. You did an initial beta test with super users that went pretty well. Then you did a big launch, and within 24 hours we were seeing some behaviour we didn’t expect. There were these batches of around 20 questions that our bot was just barely clinging on to. It was through seeing all this usage that we were able to just sit there, with our pipeline of prompts, proprietary modules, and everything else open, tweaking and levelling them up to ensure that the bot could deal with real usage. I think that’s one of the benefits of putting something into production. It can potentially become this flywheel, where the more people are asking, and the more answers they’re getting, the more you’re seeing gaps in content and what content is empowering the business. 

Tom: If I can appease anyone’s anxiety, I think that people have a lot of empathy. They know that you’re out there marketing a closed data set chatbot. They also know, as a byproduct, that it might not know the answer. And it has underwhelmed at points, but there’s a good degree of empathy out there. People are excited about this technology, and now and again we’ve both enjoyed Slack messages where it has uncovered real magic as well. So it can delight as well as underwhelm. 

Jake: It was probably around July/August that you and Andy were providing input for a vision you had for the bot, not just to power more basic Wiki-type questions, but also to be able to handle coaching support queries that may be a little further up the complexity curve, but still not complex, and still something you’d like to outsource. I believe it was you that termed it the virtual deal coach, the premise being that someone comes with an email pushback from a customer and it runs it over some kind of legal playbook. It then presents an analysis of the situation and the suggested action. It gives next steps and allows you to generate an email if you so wish. How would you describe the premise of the virtual deal coach, and how do you envision people using it?

Tom: Let’s go to the problem to be solved. For negotiating with enterprise customers, good legal teams will have playbooks designed around encouraging consistency of response and adherence to policies. We were no different in that respect. But the problem is that we were the arbiters of that playbook. At TravelPerk, we collect a lot of data on the velocity of our deals. Last June, we had stats telling us that deals were taking 12 days to close, and four of those were with us. When we did the five whys on that, we kept coming back to the idea that, as arbiters and appliers of this information, we are a bottleneck in that flow. So how do we release ourselves from that? That’s why we chose to experiment with this virtual deal coach. It's a whole lot more than a playbook, so it’s been a really interesting use case to watch our super users engage with. It might even go into the story of a clause, or into its relevance in terms of our role in the travel management sphere. But if you’ve triggered something that allows you to make a change, it will also go into the how-to of that. 

Last June, we had stats telling us that deals were taking 12 days to close, and four of those were with us. When we did the five whys on that, we kept coming back to the idea that, as arbiters and appliers of this information, we are a bottleneck in that flow. So how do we release ourselves from that?

Jake: There’s also such an opportunity, with this use case, to learn where the sales folk are at and what their gaps and areas of confusion are. I think that the potential for this kind of data, like that a particular clause has been pushed back on 120 times in the last quarter, is really powerful. 

Tom: For sure. Most legal teams have great content, but now you have the means by which you can understand what content actually matters and powers the back office of your business. Those are learnings that can have a lot of strategic impact. 

Jake: Absolutely. I think this is going to become more and more of a topic now. Our focus has obviously been to get a really strong, consistent, and reliable usage of the product. But once you’re generating that kind of data, it’s a natural corollary to contemplate what you can do with it. How can we leverage everyone’s usage of the product to make each individual customer’s usage that much more powerful and strategic. Something I wanted to spotlight on the virtual deal coach, but which applies to any part of the product, is the Chrome extension. The vision that all of these generative AI startups for end users need to be adopting is having one place where everything’s going to be done. Because, before you know it, it’s going to feel like a pain to jump into Slack. 

Tom: What I love about this technology is meeting users where they are. If you want to find me in Slack, great. But if you want to stay in Salesforce and manage your correspondence there, I’m with you as well. If you want to fill in your Google Sheet RFP, just call me as a Chrome extension. I’m always right next to you. And that’s been pretty wonderful. 

Don't miss our next webinar(s). Follow us on LinkedIn for updates.