#ICLM 2024 is almost there ๐ฅ๐ฅ๐ฅ PM if you will be in Vienna next week, Glad to catchup with the Hugging Face community there!
I would like to contribute ๐ by releasing the sixth Knowledge Vault, with 100 lectures visualized from the last 10 years of ICML from 2014 to 2024, (10 from 2024 will be included after the conference) including knowledge graphs for all the Invited Lectures and some extras, with almost 3000 topics represented using AI.
#ICLR 2024 is almost there ๐ฅ๐ฅ๐ฅ counting the days to be again in the beautiful city of Vienna participating in the The Twelfth International Conference on Learning Representations, hope to see many of the Hugging Face comunity there!
I would like to contribute ๐ by releasing the second Knowledge Vault, with 100 lectures visualized from the last 10 years of ICLR from 2014 to 2023, including knowledge graphs for all the Invited Lectures and some extras, with almost 3000 topics represented. (Of course using several AI tools including Llama3)
Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development, in which multimodal LLMs might directly convert visual designs into code implementations. In this work, we formalize this as a Design2Code task and conduct comprehensive benchmarking. Specifically, we manually curate a benchmark of 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations. We develop a suite of multimodal prompting methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We further finetune an open-source Design2Code-18B model that successfully matches the performance of Gemini Pro Vision. Both human evaluation and automatic metrics show that GPT-4V performs the best on this task compared to other models. Moreover, annotators think GPT-4V generated webpages can replace the original reference webpages in 49% of cases in terms of visual appearance and content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages are considered better than the original reference webpages. Our fine-grained break-down metrics indicate that open-source models mostly lag in recalling visual elements from the input webpages and in generating correct layout designs, while aspects like text content and coloring can be drastically improved with proper finetuning.