Cointime

Download App
iOS & Android

Google DeepMind Product Head: Every AI Product Company Should Build Its Own Benchmark Tests

According to monitoring by Dongcha Beating, Logan Kilpatrick, Senior Product Manager at Google DeepMind and head of Google AI Studio, stated on X that every company building products based on AI should establish its own benchmark tests to measure the performance of AI models. He described this as a method to make model advancements 'disproportionately beneficial to your company' and suggested that founders and business owners 'start tomorrow.' Currently, most companies rely on public leaderboards to select AI models, but these rankings measure general capabilities and are often disconnected from specific business scenarios. For instance, a company focused on contract review is primarily concerned with the accuracy of clause extraction, yet this test is not included in public benchmarks, making it difficult to assess the model's performance in this area. The benefits of building one's own benchmarks are twofold: first, it allows companies to evaluate models using their own business tasks with each update, selecting the best-performing model for their specific context rather than the one with the highest public ranking; second, it enables them to provide feedback on these test sets to model providers, encouraging continuous optimization in areas of concern. Kilpatrick noted that companies like Zapier and Sierra are already doing this, stating, 'There is a lot of alpha (excess return) to be created here.'

Comments

All Comments

Recommended for you

  • a16z Launches $2.2 Billion Crypto Fund Crypto Fund 5

    a16z has announced the establishment of its new crypto fund, 'Crypto Fund 5', with a scale of $2.2 billion. The fund will deploy capital over the next decade, covering investments across all stages of the crypto sector. a16z stated that this round of funding will focus on projects with 'real-world value', including applications in payments, financial services, and decentralized systems. The core assessment is that the current market is more suitable for building 'long-lasting infrastructure and products' rather than projects driven by short-term narratives.
  • BTC Surpasses $81,000

    Market data shows that BTC has surpassed $81,000, currently priced at $81,005.99, with a 24-hour increase of 2.79%. Due to significant market fluctuations, please ensure proper risk management.
  • South Korea Considering Trump's Hormuz Plan

    According to South Korean media Newsis: South Korea is considering Trump's Hormuz plan.
  • Russia and Ukraine Announce Unilateral Ceasefire

    On May 5, Ukrainian President Volodymyr Zelensky announced on social media that Ukraine has decided to unilaterally implement a ceasefire starting from midnight on May 6. Zelensky stated that Russian social media reported that Russia would implement a ceasefire during the Victory Day celebrations, but Ukraine has not received any formal request from Russia regarding the cessation of hostilities. Nevertheless, Ukraine has decided to implement a 'silence' starting from the specified time and will take reciprocal actions from that point. Zelensky also mentioned that it is time for Russian leaders to take real measures to end the war. His post did not specify when the Ukrainian ceasefire would end. Earlier, the Russian Ministry of Defense announced that President Putin, the Supreme Commander of the Armed Forces, decided to implement a ceasefire from May 8 to 9 to commemorate the victory of the Soviet people in the Great Patriotic War, hoping that Ukraine would follow Russia's example. (Dongxin News Agency)
  • Trump: I Made America $45 Billion in the Past 8 Months

    On May 5, Trump posted a graph on social media showing the profit curve since his administration bought Intel (INTC.US) in August 2025, captioning it: 'I made America $45 billion in the past 8 months!' (Dongxin News Agency)
  • BTC Surpasses $81,000

    Market data shows that BTC has surpassed $81,000, currently priced at $81,062.55, with a 24-hour increase of 1.29%. The market is highly volatile, so please ensure proper risk management.
  • US Bitcoin Spot ETF Sees $532 Million Net Inflow, Ethereum ETF Sees $6.13 Million Net Inflow

    On May 5, according to Farside monitoring, the US Bitcoin spot ETF saw a net inflow of $532 million yesterday, including: BlackRock IBIT: +$335 million; Fidelity FBTC: +$184 million; Morgan Stanley MSBT: +$12.2 million. The US Ethereum spot ETF had a net inflow of $6.13 million yesterday, including: BlackRock ETHA: +$54.8 million; Fidelity FETH: +$6.5 million.
  • Bitmine Stakes 192,816 ETH Again, Valued at Approximately $456.21 Million

    On May 5, according to OnchainLens monitoring, Bitmine staked 192,816 ETH seven hours ago, valued at approximately $456.21 million. Currently, its total staked ETH has reached 4,555,573, valued at around $10.69 billion.
  • Trump: Iran War May Last Another Two to Three Weeks

    On May 5, U.S. President Trump stated that the Iran war may last another two to three weeks. (Jinshi)