Falcon 180B is a large language model (LLM) with 180 billion parameters developed by the Technology Innovation Institute (TII) in Abu Dhabi. It is trained on 3.5 trillion tokens and is available for both research and commercial use. Falcon 180B is based on the Falcon series of open-source LLMs and incorporates innovations such as multi-query attention.

Key aspects of Falcon 180B:

• Performance: Falcon 180B surpasses previous open-source LLMs and rivals commercial models like Google's PaLM-2 in NLP tasks. It performs well in reasoning, coding, proficiency, and knowledge tests.

• Architecture: The model is based on the autoregressive (decoder-only) transformer architecture. It utilizes multiquery attention and multigroup attention to optimize efficiency and reduce memory consumption.

• Training: Falcon 180B was trained using 4,096 GPUs for approximately 7 million GPU hours on Amazon SageMaker.

• Open Access: The model is available on the Hugging Face Hub. The open-source nature of Falcon 180B fosters innovation and collaboration within the AI community.

• Commercial Use: Falcon 180B is available for commercial use, but the license includes restrictions and requires permission for use in a hosted service.

• Comparison: Falcon 180B has 2.5 times more parameters than Meta's LLaMA 2. Its performance is somewhere between GPT 3.5 and GPT4, depending on the benchmark used